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ABSTRACT 

This guide is intended to assist states and service 
delivery areas (SDAs) in addressing the new oversight 
responsibilities and opportunities stipulated by the Job Training 
Partnership Act (JTPA) with respect to planning a state-level 
evaluation. The following topics are covered in the individual 
chapters: the objectives and scope of JTPA evaluation (choosing to 
evaluate and reviewing JTPA evaluation materials and options); 
preliminary planning issues (assessing evaluability , building in 
utility, and understanding the organizational context); progression 
toward a JTPA evaluation plan (formulating evaluation questions and a 
research design, developing an evaluation plan, and estimating 
evaluation costs and benefits); and implementation issues (collecting 
and using data and funding and staffing a JTPA evaluation). A 
conclusion and suggestions for further reading are also provided. A 
report by Bonnie Snedeker entitled "A Report on a National/State 
Survey of Local JTPA Constituencies" is appended. (MN) 
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CONTEXT OF THIS VOLUME 

This is one in a series of volumes produced by the JTPA EVALUATION DESIGN PROJECT. 



PURPOSE AND PHILOSOPHY 

,«5 e /Iffi S? °- 1 a • Pr °i £Ct haS u £ l™ e * vel ° P a set of evaluation I00ls t^t are useful to states and local service deliverv 
areas (SDAs) .n judging the way their JTPA programs are being managed and the impact they are having. The intention 

5?,J Sh^ 5 ,V h - S r ana ? 1C a ", d man /8 e " al tool? on sound program concepts and research methods, and to de en 
S.nH inVli i^m'tS-T 10 , 11 ?- ta r' ned 1S - of P ractical and direct use in improving JTPA policies and programs at the 
and Vederal JTPA inf ° rmatl0n 15 alS0 eXpeCted 10 make a mi * ue contribution to national training policy 

It is hoped that these volumes will stimulate and support state and local evaluation efforts in JTPA, and promote more 

SSi?.? PreV1 ° US Pr °f amS With reSp f eCt t0 the issues studied and the methods us ed to investigate them An 
r, 0 ' 1 8° al is to encourage the generation of complementary information on program implementation and impact 

v I d ^ T'i a "dSDAs. Comprehensive, comparable information is essential to the development of 

a valid and reliable knowledge base for resolving problems and improving programs. It is also required for adjuring na- 
tional training strategies to changing needs and priorities at the state and local level. aujusung na 

PRODUCTS 

Consistent with this purpose and philosophy, the project has produced a set of materials to assist sta:,s and SDA< in 
?H tn ?, ir P. r °8 rams - These are to be useful in planning, designing and implementing evaluation activities. As' an 
integrated collection, each set is developed to support comprehensive evaluations over the JTPA planning cycle. 

.J^L'nn.KSi" 8 °/ th «£ materials t.° state and local users is appropriate. JTPA represents a new employment and 
Sn^^Sff *.5 y th - eXpene " C ? ° f manaEerS and the P ers P ectives of employers, but by scientific assessments 
ter n?/ addressing unemployment, poverty and other barriers to economic security. In this context, 

LnH cnnHc.v J P H^ Pr0 l ramd ? a ' S ° eXp , eCted t0 be J udged - In fact ' the Act ' s assessment requirements are more expliri 
and sophisticated than those of any employment and training legislation to date. It clearly distinguishes between monitor- 
ing activities, whose purpose is to determine compliance (such as with performance standards) and evaluation activities 
whose purpose is to determine how a program is being managed and implemented, and the kinds of effects it is havine 
««SnK% a ,«J?^ a 5nA herS - E ^ uall y. s, 8 niflcanl . new constituencies are expected to make these more rigorous 
T ■ ?■„"! ' P A u n ° w ' have this important responsibility. It is the first time in the history of employment 

and training programs that the Federal government's evaluation role has been significantly reduced. 

n.I h c! S o^ an c Ee af ^ rdsstat " and locaI , areas opportunities to influence public policy. It also requires them to assume new 
r ? spor f blll . ties - P i°8 ram evaluation is expected to become an integral part of the management of organizations 
administering, planning and delivering public training services. This is as it should be. The more information available 
at these levels, where changes in organizations can most readily be made, the more effective the management of JTPA 
programs. This project was undertaken ii that context. 

The evaluation tools produced by the project have been developed with a sensitivity to the differing needs, interests 
and resources of state and local users. They have been packaged into a single comprehensive and integrated set of volumes 
called JTPA Evaluation at the State and Local Level. The set contains planning and evaluation guides and issue papers. 
The following volumes are available in the set: 



Volume . 

.. Author 



I: Overview _ . . _ 

Project Team 

II: A Genera! Planning Guide Deborah Fddman 

III: A Guide for Process Evaluations David Grembows|{i 

III Supplement: Some Process Issues at the State Level David Grembowski 

IV: A Guide for Gross Impact Evaluations Carl Simpson 

V: A Guide for Net Impact Evaluations j my j 0 ^ nson 

VI: An Implementation Manual for Net Impact Evaluations Terry Johnson 
VII: Issues Related to Net Impact Evaluations 

A. Issues in Evaluating Cos's and Benefits Ernst Stromsdorfer 

B. The Debate Over Experimental vs. Quasi-Experimental Approaches Ann Blalock 
VIII: MIS Issues in Evaluating JTPA Davjd Grcmbowski 

HSJ,^, h A B lth c° U Wc h e r aCh °/ the d ', SCrete P roducts listed above is the responsibility of a single author, each seeks to incor- 

n7 « n r feSS '° nal PCCr reVie "'' th - C man> ' eX ? e " ent ^commendations of the advisory group, and the ideas 
and suggestions of the numerous practitioners interviewed in the process of developing tbe<e materials 



To further qualify these volumes, Volume III is accompanied by a supplement for state users This is consistent with 
the sign.f.can differences between states -nd SDAs in the kinds of process issues tha are most essential \°o S The 
I°5 ° n f im P act evaluations .s suf ciently technical, because of the statistical methods fnvdS that a practical 
manual has been written to accompany H. This guide and manual tend to be more appropriate for sS since re atS 

?&7T P h S1Z " are T ^"- ed , f0r a T aly , sis - However ' the y are useful to large SDAs and ofSS 

SDAs which may want to jointly study the net impact of their programs. Regional evaluations fSr examn e can bTverv 
productive in providing management information relevant to regional labor markets Although ?hereT5 Larate S 

Ef££h rfr 8 C0 ^ and b f u ^' th !?.£ ue is also covered the 8 ross ™P*« id n t fmpac guTdes In thi respect 
Sp«Lr t. S ,S K fr ° m "T re i ate . d out different approaches to this important element o progrfTevalua ion Abo 
iht user should be aware that the Appendix of Volume II includes A Report on a National/State Sur 4 ™ ^ Local JTPA 
Consj'tuenacs. This survey was carried out by Bonnie Snedeker, with the assistance of fcE^S^tfp^Vddi. 
tional input from practitioners to the development of the planning and process evaluation guides P 
In conclusion, several expectations have directed the development of these volumes: 

THE GUIDES 

The General Planning Guide 

This guide is to assist users in planning, funding and developing an organizational capacity to carry out process E ross 
outcome, and net .mpact evaluations and to utilize their results. Separate state and local versions are iSSffi 8 

The Evaluation Guides 
These volumes are to have the following characteristics: 

□The guides are to complement one another. 
•They are to provide information on program management and other characteristics of program implementation, which 

"Sfr S rnmP, th fc W a ay !" 7^? adm . inistrative ' managerial and service delivery policies and practices operate to affect 
outcomes, as a set of interventions separate from the program's services 

"faS m ^taa l te , n3tUre CXtent ° f err ° rS biaS " f ° r WhiCh adjustmems must be made in gross and net 
—Help explain the results of gross and net impact evaluations. 

'^oT^S?^ 0 ™!? 5011 ' 011 a88regate gr ° SS OUtCOmeS ' ° UtCOmeS differentiated bv W of service and 

~2S^ii ^ 2SS ), betWee " C,?rtain implememation modes and servi « strate gi«. ^d a broad array of client and 
—Help explain the results of net impact evaluations. 

—Suggest the more important outcomes that should be studied in net impact evaluations 

-Help sort out those aspects of implementation that may be most critical to study in process evaluations. 

•They are to provide information on net impact (the program's return on investment), which can: 
—Closely estimate the effect of the program's services on clients. 
u!E W rr h ™ esand client 8 rou P s are most important to study in broader but less rigorous gross impact studies 

IS^&tKJ^^ impIementation (particuIarly service deLry) * hich 5^25 

□The guides are to enable the user to carry oui comprehensive assessments of JTPA programs. 

"IJ ™ !1 °7 hc US2r - t0 aCQUire Several different P crs P e «ives on the same program within a particular time period- 
on program implementation, on outcomes for clients and employers and on net impact. 

'VSSSSbf^S and n w"h~y elate thCSe differem kindS ° f inf0 ™ ati0n 10 gai " a wider Standing of what 

□The guides are to describe approaches and methodologies as consistently as possible, to achieve comparability. 
•They are to define variables and relationships as similarly as possible. 

•They are to define research designs, and methods of data collection and analysis using as similar concepts as possible. 

□The guides are to draw from past resea'ch on employment and training programs, as well as seek new approaches and 
methods of specific value in evaluating JTPA at the state and local level ^pproacnes and 

'wZSiondSS'. '° ^ ^ P ° SSible the iSSUCS and measures reflected in Federal monitoring and 

'Un-ol 10 mak f. sel f ive H se ° f the results of relevant CETA studies, national studies of JTPA, and issue papers 
on JTPA evaluation by national public interest organizations in the employment and training area 

•They are to rely on the professional literature in applied social research. 



THE ISSUE PAPERS 

li^ftfiX! «°" tai " S tW °. iS u Ue P r ap - erS wh - icl l X ™ V companion pieces to the preceding volumes on net impact evalua- 
tion. The first paper on cost-benefit issues is designed to help users identify, measure and analyze relationships between 

SEf ?he a l n ,TnT"^ ary f c H 0S rr and benefit l in determining the pr °e ram s return on inv « tmen <- 

EtcELSSE? ■ , C ° nS °- f u fferent "searchstrategies associated with the net impact approach. The final volume on 
MIS ssues is to assis users in better understanding how JTPA and other employment and training manaeematth forma- 
uon systems can efficiently support the evaluation of program implementation and impact. managema " mtorma 

THE SET OF VOLUMES 

The set is integrated, but affords flexible use. The user can utilize the entire set for comprehensive evaluations over 
IZAZS & CyClC ° r l0 T r Planni " 8 P ? ri0d ' or the user can apply the information in each volume independenTl 
Syear orEnK e m Sing eValUatl ° n Md timeframeS a " d given the extent of r «° urc «- **ffwS& 



It should be understood that although evaluation products have been developed for JTPA, their basic principles and 
Srtdal progTaml ^ * * ^ ^ ^ 10 eVa ' Uate ° ther ™P'°>™n< and training Sams and 
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INTRODUCTION 



While this planning guide may be used Independently, It Is designed to 
supplement a set of evaluation guides In the series titled JTPA 
Evaluation at the S tate and Local Level . The companion volumes to this 
guide address specific JTPA evaluation research Issues. This guide 
recognizes that technical research tools are not always all that Is 
needed In carrying out a successful JTPA evaluation effort. State JTPA 
practitioners have a host of practical concerns about evaluation, 
ranging from how to promote evaluation as a worthwhile activity to how 
to hire a good consultant. The primary purpose of this guide Is to 
address these more practical concerns about planning and carrying out 
JTPA evaluation at the state level, concerns which cross-cut the 
various evaluation approaches described In the companion volumes. 

This guide begins with some thoughts about the nature, purposes and 
value of JTPA program evaluation. While students of evaluation may 
find little new here, the Ideas presented may be helpful to the 
non-specialist or to administrative decision-makers who need to know 
more about evaluation before they can support It within their 
organization. The Introductory portion of the guide Is also designed 
to familiarize the reader with the various evaluation materials 
available through the several volumes comprising JTPA Evaluation at the 
State and Local Level. From this preliminary section the reader should 
come away with a sense of how all these materials fit together and how 
they may be used to conduct various kinds of JTPA evaluations at the 
state level. 

The middle portion of this guide (Sections Two and Three) develop an 
overall planning context for carrying out JTPA evaluation. As much as 
possible, planning Issues are presented within a roughly sequential 
framework. The framework begins with an examination of what Is 
organizationally possible (what are the organizational supports for and 
constraints to evaluation) and ends with an assessment of evaluation 
costs and benefits. 

Some areas of evaluation planning are less amenable to assignment and 
discussion within a sequential framework. The final chapters of this 
guide (Section Four) are devoted to Important resource planning topics 
which deserve separate treatment. Those topics Include funding 
concerns, staffing needs and options, and data collection Issues. 

In producing this guide, the assumption Is that the potential audience 
of JTPA administrators, practitioners and evaluators Is wide-ranging In 
terms of technical background and Information needs. Such a guide 
always runs the risk of being too simplified for some and too cursory 
for others. As much as possible, this guide adheres to a middle 
course: it examines the basic evaluation planning and Implementation 
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issues within th e specific context of JTPA . but 1n honoring the 
diversity of Its readers' Interests and needs 1t does not offer a 
detailed course of action for every planning step. Readers seeking 
more Information or detail on a particular topic can refer to 
supplemental sources of information 1n the final reference section. 

Much of the background Information for the guide was collected through 
Interviews and Informal discussions with numerous federal, state and 
local JTPA practitioners, administrators, and evaluators. Almost all 
of the specific examples presented of states' evaluation experiences 
and activities are derived from these Important Informants. 



il 



SECTION 1 

AN INTRODUCTION TO JTPA EVALUATION 



These first chapters are addressed to a broad audience of JTPA administrators 
and practitioners who must decide whether or not to evaluate and how to evaluate 
JTPA. The first chapter tackles the question "Why evaluate?", setting forth some 
specific rationales for evaluating JTPA programs at the state level. In address- 
ing the concern How should we evaluate?" the second chapter describes the 
various JTPA evaluation materials and approaches contained in the set of 
volumes this guide accompanies. 
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CHAPTER 1 
CHOOSING TO EVALUATE 



What is Evaluation? 

Evaluation vs. Monitoring 
Evaluatioh Approaches 

How Does JTPA Legislation Support Evaluation? 

Federal Evaluation Responsibilities 
State Evaluatioii Responsibilities 

Why Evaluate JTPA? 
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CHAPTER 1. CHOOSING TO EVALUATE 



INTRODUCTION 

JTPA decision-makers face tough choices in allocating scarce program 
resources. While states must shoulder new oversight responsibilities, 
including evaluation of JTPA activities, JTPA administrative monies are 
restricted and no specific federal funds have been earmarked for 
evaluative purposes. As a result, evaluation activities must compete 
for recognition against other worthy program investment choices. If 
evaluation is to be accepted into the state JTPA agenda, JTPA 
administrators and policy-makers must be convinced that evaluation, as 
a program Investment, yields significant management returns. This 
chapter Introduces the concept of evaluation and argues the merits of 
incorporating evaluation activities Into JTPA programs. 

WHAT IS EVALUATION? 

This volume Is about planning and carrying out JTPA evaluation 
activities. Since "evaluation" has come to mean different things to 
different users and has often been loosely applied to an^r program 
assessment activity , we first must define the term. As It is used In 
this guide, evaluation refers to the systematic co llection , analysis 
and reporting of Information on a particular set of program activities 
and outcomes that d ecision-makers wish to know more about . 

Encompassing a variety of research methods, evaluation seeks to 
determine the e fficiency and effectiveness of a given program. 
Effectiveness concerns the extent to which a program , through various 



treatments or service Interventions has met Its Intended goals . As 
outlined In legislation, the three principle goals of JTPA are to (1) 
Increase Stable employment, (2) Increase earnings, and (3) reduce 
welfare dependency of economically disadvantaged and dislocated 
workers- In the JTPA context, then, a central question evaluation 
poses 1s "how effectively are programs contributing to changes 1n 
employment, earnings and welfare status of the Intended target group?* 

By efficiency, we mean how well a program has used available resources 
to achieve Its Intended goals. In determining the efficiency of JTPA 
program efforts, evaluation activities might focus on the various costs 
and benefits of the program and how such measures compare with those of 
other JTPA program strategies. Since 1n most state settings, JTPA 
resources are terribly limited, determining what Is an efficient use of 
those resources 1s a particularly relevant undertaking. 

While concepts of efficiency and effectiveness are Interrelated, the 
one does not necossarlly follow the other. A program may be 
tremendously efficient, yet not terribly effective, and vice-versa. 
For example, a JTPA program may be quite cost efficient In placing a 
large number of participants, but the program's true Impact 
(effectiveness) may actually be negligible;* the participants may have 
done Just as well on their own without the program. 

Evaluation and JTPA 

In measuring efficiency and effectiveness, evaluation can consider both 
JTPA program processes and outcomes . As Illustrated below, outcome 
evaluations focus on the end benefits derived from program activities; 
process evaluations focus on the activities themselves: 



Not everyone subscribes to a goal-oriented basis for evaluation. 
See Scriven, "Pros and Cons of Goal Free Evaluation.* 1 Evaluation 
Comment . Vol. 3, No. 4 (Dec, 1972), pp. 1-4. 
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OUTCOME VS. PROCESS EVALUATIONS 



Evaluation Type 



Questions Asked 



Outcome 



Did JTPA participants benefit 
from the program? 



What kind of benefits were 
derived? 



Which participants benefitted 
most? 



Process 



How was the program Implemented? 



Which program elements contrib- 
uted to or detracted from 
achievement of program goals? 



Together process and outcome evaluations can provide a wide range of 
Information to JTPA policy-makers and program staff, allowing them to 
make more Informed judgments about their programs. More specifically, 
comprehensive evaluation can Inform these decision-makers In two ways: 
decision-makers (1) can better discern to what extent major legislative 
goals for JTPA are or are not achieved and (2) can more fully 
understand how JTPA programs operate In order to better meet program 
goals and Improve compliance with performance standards. 

Evaluation vs. Monitoring 

Sometimes evaluation Is treated as If It were an elaborate extension of 
program monitoring activities. However, the evaluation process, while 
often utilizing data collected by a monitoring system, can be viewed as 
conceptually distinct from monitoring. Within the overall JTPA 
planning, management and policy framework, evaluation and monitoring 
activities should ask different questions and serve different 
purposes. As discussed above, evaluation poses questions about how 
efficiently and effectively JTPA program goals are being met or how 
they might be better met In the future. A useful evaluation permits 
decision-makers to make judgments about the value of JTPA programs (or 
particular JTPA program aspects). 
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In contrast, monitoring 1s not concerned with program Impact, but with 
program compliance, (Does this program comply with fiscal and 
programmatic requirements set forth 1n legislation, regulations and 
procedures manuals?) The ongoing assessment of program compliance Is 
an essential component In effective JTPA management and planning. 
However, Judgments of program compliance cannot substitute for 
judgments of pr ogram value : only evaluation can inform the latter. 

Evaluation Research Approaches and Methods 

Evaluation activities encompass a variety of research approaches and 
methods. Two broad categories of evaluation—Inquiry focused on 
program outcomes vs. Inquiry focused on program processes --have already 
been mentioned. The following chapter examines both process and 
outcome approaches to evaluating JTPA programs 1n more detail so that 
the reader can appreciate the several evaluation options offered In the 
companion guides to this volume. 

Because the Information needs addressed by evaluation are so varied, 
evaluation research methods are necessarily broad. JTPA evaluation 
research activities can range from exploratory case studies to more 
rigorous quasi-experimental research. Depending on the research design 
selected, the Information to be collected can run the gamut from 
strictly quantitative data (such as might be Incorporated Into the HIS) 
to the more qualitative or Impressionistic data gleaned from open-ended 
Interviews. The JTPA evaluation guides which accompany this volume 
offer several distinct and detailed research designs for examining 
different aspects of JTPA. 

Whatever evaluation approach Is used, evaluation Ideally should be a 
learning process which al lows JTPA pol Icy-makers f program 
administrators and staff to better understand and Improve their 
programs. 

HOW DOES JTPA LEGISLATION SUPPORT EVALUATION? 
The Job Training Partnership Act supports and requires program 
evaluation efforts at both the federal and at the state level . 
Congress, viewing job training programs as "an Investment In human 



capital" was particularly concerned that such programs be scrutinized 
"to determine whether the Investment has been productive." 2 Such a 
determination Is envisioned not only through mandated performance 
standards, but also through specific evaluation-related activities 
mentioned In the Act. 



Federal Evaluation Responsibilities 

At the federal level, the Act encourages and requires an array of 
evaluation-related activities, as suggested by the following excerpts 
(paraphrased In some places) from the legislation: 



• The National Committee cn Employment Policy (NCEP) shall 
evaluate performance standards as to (1) usefulness as measures 
of desired performance, (2) Impacts on choice of who Is served, 
what services are provided, and costs of such services (Sec 
106(f)) 

• The NCEP shall . . . examine and evaluate the effectiveness 
of federally-assisted employment and training programs ... and 
major federal programs which are Intended to, or potentially 
could contribute to achieving the major objectives of existing 
employment and training and related legislation. (Sec 473) 

• The Secretary shall establish a comprehensive program of 
employment and training research, utilizing the methods , 
techniques, and knowledge of the behavioral and social sciences . 
This program may Include studies concerning the development or 
Improvement of state and local employment and training programs 
. . . .(Sec. 452) 

• The Secretary shall provide for the continuing evaluation of 
all programs, activities , and projects, Including their cost 
effectiveness In achieving the Act's purposes, their Impact on 
communities and participants, their Implication for related 
programs, the extent to which they meet the needs of persons by 
age, sex, race and national origin, and the adequacy of the 
mechanism for the delivery of services . (Sec. 454) 



State Evaluation Responsibilities 

The new administrative and planning roles JTPA confers to states 
Includes the lion's share of specific program evaluation 
responsibilities. These responsibilities are encapsulated 1n the 
legislation as follows: 




"The Job Training Partnership Act, Public Law 17-300; 96 U S 
Statutes at Large 1322, Sec. 106. 
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• The Governor has general oversight responsibilities for program 
activities (Sec. 163) (Oversight Is defined 1n the Act to 
Include reviewing, monitoring and evaluating . (Sec. 103)) 

• The Governor may develop variations In performance standards to 
better evaluate programs within a specific state context. (Sec. 
106 (2)) 

• The Governors coordination and special services plan Is to 
Include evaluation of the stated JTPA activities over a two 
year period. (Sec. 121 (a)(1)) 

• The state council Is to review program operations In SDAs and the 
availability, responsiveness and adequacy of state services. The 
council Is to make recommendations to the Governor, legislature, 
elected officials, PICs and the public with respect to ways to 
Improve the effectiveness of programs . (Sec. 122 (b)(4)) 

• Each state receiving Wagner-Peyser funds *ha11 establish a 
Management Information System (HIS) . . . designed to facilitate 
the. . .analysis of programmatic and financial data for 
reporting, monitoring and evaluation purposes. (Sec. 165 (c)(2)) 

• The state council shall Identify needs . . . and . . . assesses 
the extent to which employment and training, vocational 
education, rehabilitation services, public assistance, economic 
development and other programs and services represent a 
consistent. Integrated and coordinated approach to meeting 
such needs. (Sec. 122 (b)(7)) 

The above excerpts suggest that the state f s oversight responsibilities 
extend far beyond the narrow realm of compliance monitoring. Rather, 
the legislation envisions the state playing a broad and significant 
evaluation role In examining the effectiveness and efficiency of not 
only its own administrative activities, but of JTPA programs In general. 

WHY EVALUATE JTPA? 

We have described the general purpose behind any program evaluation: 
It Is to learn more about the efficiency and effectiveness of a program 
so that it may be Improved. We have also touched on some specific 
legislative supports to conducting JTPA evaluation at the state level. 
To complete the answer to "why evaluate JTPA?" we conclude with five 
more specific arguments for evaluating JTPA. 



1. Evaluation as a Mechanism for Accountability 

With shrinking public resources have come Increasing demands for 
viable methods to ensure that program funds are being wisely 
spent. A program's worth must be demonstrated not only to 
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• The state council 1s to review program operations In SDAs and the 
availability, responsiveness and adequacy of state services. The 
council Is to make recommendations to the Governor, legislature, 
elected officials, PICs and the public with respect to ways to 
Improve the effectiveness of programs . (Sec. 122 (b)(4)) 

• Each state receiving Wagner-Peyser funds *ha11 establish a 
Management Information System (HIS) . . . designed to facilitate 
the. . .analysis of programmatic and financial data for 
reporting, monitoring and evaluation purposes. (Sec. 165 (c)(2)) 

• The state council shall Identify needs . . . and . . . assesses 
the extent to which employment and training, vocational 
education, rehabilitation services, public assistance, economic 
development and other programs and services represent a 
consistent. Integrated and coordinated approach to meeting 
such needs. (Sec. 122 (b)(7)) 

The above excerpts suggest that the state's oversight responsibilities 
extend far beyond the narrow realm of compliance monitoring. Rather, 
the legislation envisions the state playing a broad and significant 
evaluation role In examining the effectiveness and efficiency of not 
only its own administrative activities, but of JTPA programs In general. 



WHY EVALUATE JTPA? 

We have described the general purpose behind any program evaluation: 
It Is to learn more about the efficiency and effectiveness of a program 
so that It may be Improved. We have also touched on some specific 
legislative supports to conducting JTPA evaluation at the state level. 
To complete the answer to "why evaluate JTPA?" we conclude with five 
more specific arguments for evaluating JTPA. 



1. Evaluation as a Mechanism for Accountability 

With shrinking public resources have come Increasing demands for 
viable methods to ensure that program funds are being wisely 
spent. A program's worth must be demonstrated not only to 
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This kind of Information sharing 1s particularly Important 1n a 
complex, multi-layered service delivery system as 1s found 1n the 
more decentralized JTPA programs (I.e., where service delivery 1s 
contracted out). For example, the rich and complex results of a 
Process evaluation may allow PIC members, elected officials, 
state administrators and the public to more directly grasp the 
complexities of effectively managing an employment and training 
program. Those outside service delivery can gain a better 
appreciation for the difficulties In delivering JTPA services, 
given the resource constraints, coordination demands and 
organizational obstacles many service providers face. Similarly, 
evaluation offers state administrative staff and SDAs an 
opportunity to communicate more effectively with each other about 
their separate concerns. 

Evaluation as a Tool for Moving Beyond Performance 
Standards 

From Its Inception, JTPA has focused attention on one type of 
performance assessment: performance standards. Since the 
standards are mandated and are to be uniformly applied across 
states and their SDAs (unless states choose to develop their own 
regression model for performance standards) , why be concerned 
about other assessment measures which are not so explicitly 
called for In legislation? Evaluation tools are a necessary 
complement to performance measures for several reasons: 

• Evaluation Kelps Explain Performance . 

Consistently low performance outcomes or Inconsistent 
outcomes may clue us In that there Is a problem, but tell us 
little about what Is Influencing such performance. 
Performance standards do not tell us what Is or Is not an 
effective program element. To answer these kinds of 
questions, we may use a process evaluation to systematically 
examine specific program factors. 

• Evaluation Looks at Distributive Outcomes . 

High or low performance outcomes may mask other less obvious 
distributive outcomes for clients. Who Is really being 
served by JTPA? Is It the Intended target group? Are 
clients receiving truly beneficial services and placements? 
Evaluation can directly address these distributive Issues; 
performance measurements can do so only In a limited fashion. 

• Evaluation Measures Program Impacts . 

Performance measures alone cannot answer the Important 
question "Did the program have an Impact In giving people 
durable jobs, Increasing their earnings and reducing their 
vulnerability to poverty?" While a given program may boast 
a 70% placement rate, we have no Idea If the program's 
efforts were truly responsible for those placements or 
whether, In fact, participants would have gotten the same 
jobs even If they had not participated In JTPA. Performance 
outcomes need to be supplemented with other evaluation 
techniques that help sort out extraneous Influences from the 
true effects of the program Itself. In some Instances, 
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evaluation may reveal that a program with low performance 
measures Is still very effective because 1t significantly 
Impacts a target group of dlff 1cult-to-serve clients who, 
without the program, would otherwise not have been 
successfully trained and placed. 

CONCLUSION 

Congress Intended JTPA to be a "performance-driven system" In which 
the program's measured accomplishments In training and placing 
participants would be the hallmark of program success. 

In such a decentralized federal setting, it makes sense to develop 
national performance standards and reporting requirements to ensure 
a measure of program accountability to federal authorities. But 
successful compliance with one's assigned numerical goals Is only 
one source for judging the value of a program. Evaluation offers 
other Important sources of Information which help JTPA 
decision-makers to see the complexity of the program and to make 
more accurate assessments of Its true Impact on participants. Armed 
with such Information, those decision-makers are then In a better 
position to develop strategies for further program Improvement. 

What evaluation course make sense for state JTPA decision-makers to 
pursue In order to capture the benefits of evaluation described 1n 
the preceding pages? The following chapter delves further Into the 
specific kinds of evaluation approaches and options available 
through additional guides In the set Evaluating JTPA at the State 
and Local Level. 
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CHAPTER 2 

REVIEWING JTPA EVALUATION MATERIALS AND 

OPTIONS 



What Evaluation Materials Are Available? 
What Is a Gross Impact Evaluation? 
What Is a Net Impact Evaluation? 
What Is a Process Evaluation? 
How do These Evaluation Approaches Complement Each Other? 



CHAPTER 2. JTPA EVALUATION MATERIALS AND OPTIONS 



INTRODUCTION 

With the passage of JTPA in 1982, Congress created a new legislative 
context for planning and Implementing this country's employment and 
training programs. The new context Includes a much enhanced 
administrative and planning role for state government. That role now 
encompasses the setting up of new service delivery areas (SDAs), the 
planning and overseeing of special coordination efforts, and the 
administration of special programs. 

Additionally, JTPA has created new evaluation roles and 
responsibilities at the state level. Previously, the federal 
government formulated evaluation policy, funded new evaluation research 
efforts and disseminated findings. Under JTPA, states now must take on 
new oversight responsibilities, having relatively little experience In 
evaluation policy-making, design and Implementation. The materials 
described here are part of a research effort to assist states 1n 
carrying out these new roles and responsibilities. 

THE JTPA EVALUATION DESIGN PROJECT 

This planning guide Is one In a series of related evaluation materials 
produced by the JTPA Evaluation Design Project. 3 In this 
chapter we will briefly describe the Project's purposes and 
orientation, present the various materials available to state users 
through the project and outline how state administrators, planners and 
policy-makers can effectively use these materials. 



For a synopsis of the Project and Its funders and participants, see 
the Preface. 
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A primary purpose of the project 1s to create evaluation materials 
which are useful to states and SDAs in planning and carrying out JTPA 
evaluation activities. A secondary purpose 1s to develop several model 
evaluation strategies which, when applied across states and SDAs, can 
produce comparable Information. If states, for example, use a 
consistent research strategy to assess the Impact of JTPA service*: on 
clients' future employment and earnings, then these findings are more 
likely to have broader significance, Informing policy-makers at the 
federal, as well as the state and local levels. 

WHAT EVALUATION MATERIALS ARE AVAILABLE? 

In order to meet different users' needs, the evaluation materials 
developed by the project consist of a set of complementary volumes on 
JTPA evaluation entitled JTPA Evaluation at the State and Local Level . 
These volumes can be used Independently or 1n conjunction with each 
other. The set of materials described 1n this chapter Includes the 
following volumes: 



Volume I: 
Volume II: 
Volume III: 
Volume IV: 
Volume V: 
Volume VI 
Volume VII: 



Volume VIII: 



Overview 

A General Planning Guide (state or local version) 

A Guide for Process Evaluations 

A Guide for Gross Impact Evaluations 

A Guide for Net Impact Evaluations 

An Implementation Manual for Net Impact Evaluations 

Issues Related to Net Impact Evaluation 

a. Issues 1n Evaluating Costs and Benefits 

b. The Debate Over Experimental Vs. Quasi- 
Experimental Design 

MIS Issues 1n Evaluating JTPA 



This set of volumes 1s designed to offer state and local level users a 
fairly selective, yet comprehensive menu of technical assistance 
products to meet a variety of evaluation needs. Taken together, these 
products support comprehensive evaluations over the JTPA biennial 
planning cycle. However, users may also wish to selectively choose 
from this menu 1n order to meet a state's particular evaluation 
Interests, needs and resources. To give a sense of the utility and 
scope of these materials, the various volumes are briefly described as 
follows. 
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Overview (Volume I) 

This summary volume outlines the materials comprising the set of 
volumes JTPA Evaluation at the State and Local level . m condensed 
form, 1t covers the specific evaluation questions, research Issues and 
methodological concerns addressed 1n each of the companion guides 1n 
the total series. 

A General Planning Guide (Volume II) 

This volume provides an overview of the various evaluation tools 
available 1n Volumes III through VII and how these tools may be used 1n 
a complementary fashion. Additionally, the volume focuses on practical 
planning and Implementation Issues that cross-cut various evaluation 
designs, such as how to develop the organizational capability for 
evaluation and how evaluation activities at the state level might be 
planned, funded and carried out. A state and a local version of this 
guide are available. 

A Guide for Process Evaluations (Volume III) 

A Guide for Gross Impact Evaluations (Volume IV) 

A Guide for Net Impact Evaluations (Volume V) 

These analysis guides present three distinct approaches and related 
methodologies for analyzing and carrying out JTPA nrogram evaluation. 
(A specific discussion of the uses and complementary Interaction of 
these designs follows later 1n this chapter.) Each guide contains 
these components: 

1. A framework for analyzing either JTPA program activities 
(process evaluation) or outcomes (net and gross impact 
evaluations), including the specific kinds of evaluation issues 
each approach addresses and the kinds of variables, 
measurements and data sources each approach requires. 

2. A discussion of research methodology, including: 

• A recommended research design approach for answering a key 
set of evaluation questions. 

• A description of data collection and analysis methods 
covering potential pitfalls, problems and possibilities, 
including recommendations for the use of MIS data elements 
and other data bases, where relevant. 

3. An appendix to the guide providing additional references 
and/or technical information relating to each approach. 

Each of these analysis guides may be used independently, in conjunction 
with each other, or with Volume VII and VIII. To the degree possible, 
the guides present Information in a straight-forward, non-technical 
fashion 1 * in an effort to make the presentation accessible to a wide 



Because of the research design requirements of the net impact 
approach, the net impact evaluation guide, of necessity, contains 
more technical information than the other two guides. 
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audience of potential users. The development of these products has 
also been Influenced by project concerns that tha materials be attuned 
to evaluation Issues of greatest Interest to state and local users, be 
realistically Implementable 1n terms of research cost and complexity, 
and be committed to scientifically sound research strategies. 

An Implementation Manual for Net Impact Evaluations 
(Volume VI) 

The volume on net Impact evaluations 1s sufficiently technical, because 
of the statistical methods Involved, that this practical manual 
accompanies It. 

Issues Related to Net Impact Evaluation (Volume VII) 

• Issues in Evaluating Costs and Benefits 

• The Debate over Experimental vs. Quasi-Experimental 
Design 

This first Issue paper describes the rationale and procedures for 
estimating JTPA program costs, showing how costs and benefits are 
related In a human capital Investment framework. The second paper 
examines the pros and cons of two different net Impact research 
strategies. 

MIS Issues in Evaluating JTPA (Volume VIII) 

This Issue paper Is designed to assist users In better understanding 
how JTPA and other employment and training management Information 
systems can efficiently support evaluation. 



Which of these analysis tools will best serve the evaluation needs and 
capabilities of an Individual state? Given various resource 
constraints, what kind of evaluation approach should take priority? 
The answer depends In large measure on the kinds of policy priorities 
your state has established and the evaluation questions of greatest 
relevance to JTPA planners and policy-makers. 



The remainder of the chapter outlines the principle features of the 
three major evaluation approaches, the strengths and limitations of 
each approach and the key questions about JTPA programs each 
addresses. We begin with a look at evaluations which focus on outcomes. 



WHAT IS A GROSS IMPACT EVALUATION? 

In general, evaluations of the outcomes of a program are designed to 
analyze various short-term and long-term accomplishments In the context 
of the program 1 s stated goal s . Outcome evaluations , as the name 
Implies, focus on the end products of the program-- in this case, 
measures of those employed, their wages, their status with respect to 
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the welfare system, and program costs. (Other outcomes can be measured 
too, such as additional client outcomes, employer outcomes, and more 
general societal outcomes, such as taxpayer dollars saved.) Program 
outcome measures, taken by themselves, (without comparing them to 
outcomes for similar Individuals who do not receive JTPA-lIke services) 
can be considered gross outcomes. 

gross Impact evaluation provides a systematic way to describe 
post-program outcomes and to analyze how service delivery alternatives 
Influence them. The gross Impact approach can be used at the state 
level to study outcomes across SDAs or at the local level to study 
outcomes within a single SDA. 5 The distinctive feature of a gross 
Impact evaluation Is Its exclusive focus on outcomes related to program 
participants: there Is no comparison or control group of 
non-participants to provide a yardstick against which overall program 
outcomes may be assessed. 

Because no untreated control group Is utilized, the gross impact 
evaluation cannot explain participant outcomes In terms of the 
program's efficacy. In using this approach to evaluate JTPA programs, 
we do not know to what extent the outcomes are t.ie product of other 
external Influences, such as changes In the economy, varying client 
characteristics, client use of non-JTPA training and educational 
programs or chance. In other words, we cannot differentiate between 
Impacts caused by the program and results that would have occurred 1n 
Its absence. 

While unable to address the singular Impact of JTPA programs, the gross 
Impact approach offers some distinct advantages: 

• The resec. .h design may be less complex and easier to Implement 
than a net Impact design. 



Providing the SDA's client base Is large enough to create a 
sufficient study sample. Volume IV discusses sample size and other 
research considerations for state and local users. 



ERIC 



18 

28 



The approach offers a fairly quick turnaround time for Information 
results. 



• A wide range of key variables may be measured, allowing for a 
richer understanding of the program's performance. 

• The relative Impacts of different service strategies may be 
assessed. 

Host Importantly, In addition to the above-mentioned characteristics, 
the gross impact evaluation provides a framework for answering key 
questions about service delivery strategies, program types and employer 
and trainee post-program experiences with JTPA. In turn, these answers 
may Inform policy-making and program planning at both the state and 
local level. Some of the central questions a gross Impact evaluation 
can address are framed below: 



GROSS IMPACT EVALUATION QUESTIONS 



General FoIIow-Ud: What Is the overall picture of participant 
employment, wages, and welfare status at some distinct time 
period after termination? How does the picture change at 
three months, six months, nine months? 

Employer Outcomes: How do JTPA trainees Impact employers? To 
what degree does employer participation In JTPA raise or lower 
company turnover rate, affect training time, supervision or 
hiring? 

Compariso n of Treatments :* Which treatment strategies 
(e.g., long-term vs. short-term, OJT vs. classroom training) 
have more positive outcomes relative to other treatment 
strategies? 

Comparison of Treatments Across Different Client Groups : * 
Which treatment strategies are most effective for different 
client sub-populations, relative to other treatment strategies? 

Quality of Placements: Do post-program jobs for JTPA clients 
resemble primary, as opposed to lower quality, secondary labor 
market positions? Are positions training-related? 

Notes The methodology used 1n the guide to answer this 
question Is referred to as "differential gross 
Impact analysis." 
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While a gross Impact evaluation can answer questions about the relative 
merits of different JTPA program components, to find out about the 
true effectiveness of JTPA we have to turn to a different kind of 
outcome evaluation, the net impact evaluation , 

WHAT IS A NET IMPACT EVALUATION? 

In contrast to gross impact, a net Impact evaluation attempts to sort 
out specific program impacts from other Influencing factors. A net 
Impact evaluation more precisely answers the question "was the program 
effective?" by analyzing the extent to which outcomes were due 
specifically to program treatments rather than to other factors, such 
as participant characteristics or the environment In which the program 
operates. 

Of necessity, a net Impact evaluation approach requires a complex 
theoretical base and may require a larger sample size than the gross 
Impact evaluation. The hallmark of the net Impact research design Is 
the Inclusion of a comparison group of non-participants whose 
performance establishes a baseline against which JTPA client outcomes 
may be judged. The question, then, really becomes "Do JTPA clients do 
significantly better in the labor market than non-participants with 
similar economic and educational profiles?" 

Some potential limitations of the net Impact evaluation are its greater 
design complexity and special data requirements: data elements 
required from non-JTPA sources may be difficult to access or 
unavailable. In most cases, the evaluation will be limited to the 
study of a small set of key variables and outcomes. 6 However, as a 
balance to these limitations, the net Impact design offers a powerful 
evaluation tool--a tool that allows us to Identify more direct causal 



The net Impact approach developed 1n this series uses a 
quasi -experimental design, as opposed to a true experimental 
design. Comparison group members are statistically matched to the 
experimental group of self-selected JTPA participants rather than 
all participants being randomly assigned to either group. Thus, 
this net Impact approach must make additional validity assumptions 
about what 1s being measured. These additional assumptions may be 
viewed as a limitation imbedded within the design. 
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links between JTPA service and client outcomes, thus permitting 
stronger policy conclusions. 

In assessing the effectiveness of JTPA, a state may wish to 
know not only how effective JTPA 1s 1n general, but also how 
effective different program strategies are for various client 
subgroups. Some additional questions which a net Impact 
evaluation of JTPA can address Include: 



NET IMPACT EVALUATION QUESTIONS 



• Which program types have a greater Impact on 
client earnings? 

• Is long-term training more effective than 
short-term? 

• Are multi-strategy program approaches more 
likely to have a greater Impact than single 
strategy programs? 

• Do some client groups benefit more from 
certain types of training than other client 
groups? 



WHAT IS A PROCESS EVALUATION? 

By definition, outcome evaluations tell us primarily about program 
Jesuits. Examination of the factors which contribute to or help 
explain tnose results Is more the province of process evaluations . 7 

Is a JTPA program underperformlng because of the services provided to 
clients or because of the way services are delivered? In order to 
provide Insights Into why_ a program Is achieving particular results, a 
process evaluation Illuminates the organizational manner In which the 
program Is carried out. How are services assigned to target 
populations? How are client flows organized? How are program 
functions carried out and Inter-program coordination accomplished? in 
responding to these sorts of questions, a process evaluation can reveal 
Important Influences that program Implementation factors have on 



7 evaluat°1on timeS referred to as im P 1ef "entat1on studies or formative 
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program outcomes. 



For example, how an SDA organizes Its outreach and Intake procedures 
may Intentionally or unintentionally affect which kinds of clients 
enter the JTPA system and what kinds of services they receive— the 
selection procedures ultimately shaping employment, earnings and 
welfare savings. At the state level, a JTPA process evaluation will 
attempt to sift out those administrative and coordination arrangements 
which appear to have the most Influence on the nature and quality of 
service provision, Identifying which arrangements are contributing to 
goal achievement and which are Inhibiting It. 

With the possible exception of single SDA states, JTPA Implementation 
encompasses two separate but Interrelated organizational levels, the 
state administrative level and the local service delivery level. 
Therefore, process evaluation at the two levels will be distinct from 
each other (although state decision-makers may be concerned with 
assessing Implementation at both levels), posing different questions 
about Implementation of JTPA. Some of the key questions posed by the 
state level and SDA level process evaluations are framed below: 



PROCESS EVALUATION QUESTIONS 



State Level Process Evaluation 

How are state JTPA policies 
being formulated and Implem- 
ented? 



How are state policies and 
procedures affecting JTPA 
service delivery? 

How might communication and 
coordination between state 
agencies, states and SDAs, 
and states and federal JTPA 
administrators be Improved? 



SDA Level Process Evaluation 

What are the service goals 
of the SDA? Do these goals 
mesh with state employment 
and training goals? With 
JTPA goals? 

Hew are service delivery 
arrangements affecting who 
receives services? 

Are certain service delivery 
arrangements supporting or 
Inhibiting achievement of 
JTPA goals or particular 
state and SDA goals? 



In answering these kinds of questions about JTPA organizational 
arrangements, the process evaluation must rely on a number of data 
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sources, Including less quantifiable data gathered from observation and 
Interviews. As a result, the Inferences drawn from the data may be 
more subjective 1n nature. However, providing the evaluators have 
sufficient understanding of the given JTPA program context and 
sufficient neutrality and distance from the program Itself, process 
Information can provide valuable Insights Into the program's strengths 
and weaknesses. 



HOW DO THESE THREE APPROACHES COMPLEMENT ONE ANOTHER? 
While each of these evaluation approaches have been treated thus far as 
free-standing, they are designed to be used In a complementary fashion 
over the JTPA biennial planning cycle. In combination, these various 
evaluation approaches offer more powerful analytic tools for 
understanding and Improving JTPA. (For an overview of the three 
evaluation types, see the following page.) Here are some ways In which 
these tools can Interact: 



Process Evaluations Hake Impact Findings More Meaningful . If 
outcome measures suggest a particular program performed well , 
administrators and planners will want to know what factors 
contributed to the program's success. The outcome measures of 
net and gross Impact studies are of limited utility If factors 
contributing to that success cannot be described or duplicated 
by others. In helping to answer the guestlon H what worked?", a 
process evaluation fills an Important Information gap. 
Similarly, where outcome measures Indicate poor program 
performance, a process evaluation can help decipher the reasons 
for Inadequate performance and suggest changes for Improvement. 



Gross Impact Evaluation Hay Help to Focus or Enrich a Process 
Evaluation . Often resource and time limitations dictate that a 
process evaluation be limited In scope to those elements or 
processes which are most likely to yield Interesting and useful 
Information. Data collected on gross outcomes may help pinpoint 
the focus of a process evaluation more precisely to the most 
productive areas of Inquiry. 



Gross and Net Impact Evaluations Kay Interact and Inform Each 
Other 1n Important Ways . Although both gross and net 
evaluation approaches may be used to compare outcomes for 
different client subgroups and different service categories, the 
gross Impact evaluation allows for a more wide-ranging analysis 
of specific JTPA treatments and policies not possible (because 
of time, resource or data constraints) with the net Impact 
approach. The quick turnaround descriptions furnished by the 
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OVERVIEW OF PROCESS, GROSS IMPACT AND NET IMPACT EVALUATIONS 



PROCESS EVALUATION 



A tool for studying the way JTPA is 
being implemented, and the influence 
implementation processes are having 
on client outcomes. 



QUESTIONS ASKED: How are the major 
implementation characteristics of 
the program (which are expected to 
produce positive c*j tccrr^s ) influ- 
encing outcomes? Are they working 
as planned? 

BENEFITS OF THE INFORMATION: Often 
it is the program's features that 
are affecting outcomes more than 
the services provided, process 
information helps the user pinpoint 
the differential effects of service 
treatments vs. the way the program 
is being carried out. 

LIMITATIONS OF THE INFORMATION: 
Process information is often diff- 
icult to quantify, and therefore 
the inferences are more subjective. 
Nevertheless, significant clues to 
relationships between processes and 
and outcomes are possible. 



DISTINCTIVE FEATURE: The user can 
identify those elements of implem- 
entation that are contributing to 
goal achievement, or inhabiting it. 



GROSS IMPACT EVALUATION 



A tool for studying gross outcomes 
for clients and employers: For 
all clients; for different client 
groups; for clients receiving 
different service interventions. 



QUESTIONS ASKED: What are post 
program outcomes for clients (and 
employers) who experience JTPA? 
Wnai Service strategies produce 
the most positive outcomes relative 
to all other strategies? 

BENEFITS OF THE INFORMATION: States 
and SDAs can track the kinds of out- 
comes that characterize different 
groups given different services, 
without collecting information on a 
comparison group. 



LIMITATIONS OF THE INFORMATION: 
In interpreting the information, we 
can not attribute any of the out- 
comes to the program itself. We 
can only say that the outcomes are 
occurring, due to a potential range 
of influences, one of which is the 
program. 

DISTINCTIVE FEATURE: The user can 
obtain information on a rich range 
of outcomes for both clients and 
JTPA employers, not available 
through the net impact model. 



NET IMPACT EVALUATION 



A tool for studying the net impact of 
the program on clients: For all clients; 
for clients receiving different service 
interventions; for different client 
groups— utilizing a comparison group 
to control for non-program influences 
on outcomes. 

QUESTIONS ASKED: Df the key outcomes 
in the legislation, which outcomes 
can be attributed to JTPA, rather than 
to other influences or to chance? What 
service strategies are most effective 
for which subgroups of clients? 

BENEFITS DF THE INFORMATION: The user 
can sort out which outcomes are due to 
the service interventions, rather than 
to other cause:. Consequently, the 
user has a measure of return on the 
investment. 



LIMITATIONS OF THE INFORMATION: 
Because of data availability, only a 
small set of key outcomes can be 
studied. 



DISTINCTIVE FEATURE: Policy makers can 
more truly judge the effectiveness of 
JTPA programs and service strategies. 
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gross Impact approach may help provide clues suggesting the reasons 
for particular net Impact findings. 

The wider array of outcome measures 1n the gross Impact approach may 
be merged with net Impact data and used to help explain net Impact 
findings. For example, gross outcome measures may Include 
Information on quality of placement and the job satisfaction of the 
JTPA client. Linking such measures to net Impact findings may help 
answer questions about the relationship between quality of placement 
and long-term earnings or job retention. 

CONCLUSION 

The purpose of this chapter has been to Introduce the various 
evaluation tools contained In the set of guides entitled JTPA 
Evaluatio n at the State and Local Level . While each guide may be 
used Independently, the guides are designed to complement one 
another; taken as a totality they offer a comprehensive view of JTPA 
evaluation Issues and approaches. In particular, the three major 
evaluation approaches designated as net Impact , gross Impact and 
Process can Interact and Inform each other In significant ways. 
Setting aside for a moment concerns about choosing an evaluation 
approach (or approaches), we now examine some preliminary evaluation 
Issues which cross-cut the various approaches. 
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SECTION 2 
PRELIMINARY PLANNING ISSUES 



The following chapters cover some preliminary planning issues to be considered 
early on in a JTPA evaluation planning effort. While these issues are presented 
within an overall temporal framework, they do not translate easily into a set of 
discrete sequential planning steps to be set down in a guide. Rather, the plan- 
ning issues, which for clarity's sake are discussed here under separate category 
headings, in actual practice blend and overlap extensively with one another. 
These early evaluation considerations have long-range implications for the plan- 
ning and implementation work that occurs at later stages. While these chapters 
do not offer a defined set of planning steps, they contain numerous strategy 
considerations, check-lists, and suggestions for beginning the JTPA evaluation 
process. 
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CHAPTER 3 

ASSESSING EVALUABILITY AND BUILDING IN UTILITY 

What Kind of Evaluation is Feasible? 

Evaluability Assessment 
Factors Affecting Evaluability 
Improving Program Evaluability 

Will the Evaluation be Utilized? 

Barriers to Utilization 
Increasing Evaluation's Utility and Utilization 



CHAPTER 3. 



ASSESSING E VALUAB I L I T Y AND BUILDING IN UTILITY 



INTRODUCTION 

Even before specific evaluation questions are delineated or an 
evaluation approach settled upon, some Important preliminary planning 
Issues must be considered. This preliminary planning work revolves 
around three Interrelated questions concerning the setting In which the 
evaluation occurs: 

• What kind of evaluation Is feasible? 

• To what extent will the evaluation be utilized? 

• How does the organizational context impact evaluation? 

How these questions are dealt with will have long-range consequences 
for the Implementation of the evaluation and Its ultimate Integrity as 
a useful planning, policy and management tool within JTPA. This 
chapter examines the first two questions; the following chapter 
continues with the third question. 

WHAT KIND OF EVALUATION IS FEASIBLE? 

Before fully embarking on an evaluation plan, evaluators should 
conslde the feasibility of evaluating a particular JTPA program. Are 
some kinds of evaluation efforts more likely to succeed than others? 
Is the timing appropriate, or would an evaluation yield better results 
at a later date? 

To answer these kinds of questions, Rutman and others suggest that 
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evaluation planners begin with an "evaluabllUy assessment" of the 
program 1n question. Such a preliminary assessment will help an 
organization to: 



Define the appropriate scope and timing for an evaluation 

Avoid wasting time and planning efforts that will not produce 
useful results 

Identify barriers to evaluation that need to be removed before 
evaluation can take place 1 

Lay the groundwork for doing further evaluation planning when 
circumstances are more conducive to such efforts 



Rutman has outlined In detail step-by-step procedures for assessing a 
program's evaluabllUy. Some or the major points he and others have 
made are summarized here In terms of (1) technical factors; (2) nrogram 
features; and (3) organizational factors affecting evaluabllUy. 



Technical Features Affecting Evaluability 

Some of the most obvious barriers to doing solid, useful evaluation of 
JTPA programs are largely technical In nature. In later chapters, 
several of these technical Issues concerning funding, staffing and 
managing JTPA evaluation efforts will be presented In greater detail. 
For clarity's sake, these technical concerns, as they Touch on program 
evaluabllUy, are briefly mentioned as follows: 

• Financial Constraints : Are thera sufficient funds to ensure 
the evaluation efforts successful completion? If not, can 
additional funds be obtained within an acceptable timeframe? A 
scaled down, but well- supported evaluation effort, providing 
quality Information In a few key areas may prove to be the most 
useful Interim option. (Chapter 9 discusses JTPA evaluation 
funding strategies further.) 

• Staffing Constraints : In-house staffing of an evaluation 
effort Is one way to overcome financial constraints, but If 
staff resources are stretched thin, this strategy may end up 
compromising the quality and usefulness of the evaluation. 



See , for exampl e , Leonard Rutman , Planning Useful Evaluations: 
Evaluabll Uy Assessment . Sage Library of Social Research, Vol 96 
(Severely Hills, CA: Sage Publications, 1980); Joseph S. Wholey, 

Evaluation: Promise and Performance (Washington, D.C.: The Urban 

Institute, 1979); and Richard E. Schmidt, John Scanlon, and James 
B - Bel1 » EvaluabllUy Assessment (Rockvllle, HD: Project SHARE 
O DHEW no. : 05-76-730, 1979). 
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Alternatively, such constraints may encourage "creative 
leveraging" of both governmental and community resources 
heretofore untapped. (Overcoming staffing constraints 1s 
treated separately 1n Chapter 10.) 

Evaluation Timeframe: To be most useful, evaluation must be 
timely In answering the questions of chief Interest to JTPA 
program administrators and policy-makers. If the timeframe for 
collecting and analyzing data Is too liberal, evaluation 
findings may become stale and less relevant to decision-makers. 

Data Collection Problems: Insufficient data or Inaccessible 
data may also delimit the nature and scope of an evaluation 
effort. (JtPA data collection Issues are detailed In Chapter 



Program Features Affecting Evaluability 

Another set of factors affecting evaluability has to do with the 
contours of the program Itself. While there may be no substantial 
technical barriers to conducting an evaluation, an employment and 
training program Itself may exhibit certain characteristics which make 
evaluation outcomes more difficult to Interpret and utilize 
effectively. Typically, a process (or Implementation) study may be 
necessary to elucidate such features before larger-scale outcome 
evaluations are considered. Some of the characteristics affecting 
evaluability are as follows: 



Changing or Unfocused Goals : Explicit program goals provide a 
predetermined standard against which program processes and 
accomplishments can be measured. when an employment and 
training program's goals are unfocused or constantly changing, 
the task of evaluation Is more difficult: how do you measure 
your achievements If you are not clear about what It Is you are 
trying to achieve? 

Multiple and Conflicting Goals : Program goals may be 
well-defined, but Inconsistent with each other, complicating 
the task of evaluation. For example, the goal of achieving a 
high placement rate at a low cost per placement often conflicts 
with other goals, such as significantly developing skill levels 
of participants or long-term retention of trainees 1n their 
placements. Such goal conflicts are Inherent to many JTPA 
programs; the Issue Is not that of completely eliminating such 
conflicts (an impossible task!), but making the evaluation 
approach as sensitive as possible to such constraints on 
program outcomes. (A process evaluation may be needed to sort 
out how different program activities are supporting conflicting 
goals.) 



Variable Service Provision Strategies : When programs encompass 
numerous service provision strategies (as Is the case 1n many 
JTPA program settings) or change strategies m1d-stream f the 
task of evaluating becomes more challenging. The less uniform 
the overall treatments given, the more complicated the task of 
adequately accounting for program Impacts. 

Small Program Size : The size of the program may also shape the 
nature and scope of evaluation. In the case of smaller, 
special 1TPA projects or programs (for dislocated workers, 
older workers, etc.) Impact findings may be of limited 
usefulness due to small sample problems or cost 
Inefficiencies. (A fuller discussion of sample size 
requirements Is found In the gross and net Impact designs 
presented In Volumes IV and V.) 



Organizational Factors Affecting Evaluability 

Organizational factors often present the least tangible, but most 

powerful barriers to useful evaluation. Because of the central 

Influence they have over evaluation activities, organizational concerns 

will be examined In greater detail 1n the following chapter. Some 

common organizational factors Impacting evaluability are encapsulated 

below: 



Staffing Problems : When a program Is plagued with low staff 
morale or high turnover, something Is clearly wrong, but an 
evaluation may not help. Evaluation activities may create 
added burdens for the staff which they cannot handle. 
Effective staff are crucial In the operation of any social 
service program. An organization with serious staff problems 
will probably first need to focus Its energy on rectifying 
those problems before being able to utilize broad evaluation 
findings . 

The History of Previous Evaluation : Have previous evaluations 
been done? If so, how have they been used? If the results 
have been Ignored, Is there any evidence to suggest that a new 
evaluation will receive any better reception? Alternatively, 
have evaluations been used to punish or undermine certain 
factions or personnel within the organization? If so, the 
credlbl llty and usefulness of the new evaluation may be 
questioned and staff cooperation lost. Evaluation planners 
will have to develop some Initial strategies to build up trust 
and credibility. 



33 

41 



• Hidden Agendas: In some cases, the sponsor of the evaluation 
1s not truly committed to an open Inquiry Into program 
operations from which the program can learn or Improve. 
Instead the sponsor wants to use the evaluation to support a 
preconceived notion of the program as worthwhile or not 
worthwhile. 

• Financial Difficulties : When a program Is struggling to stay 
afloat financially, the utility of an evaluation Is often 
severely curtailed. Administrative energy Is necessarily 
focused on program survival rather than program Improvement. 
The program may be able to take better advantage of evaluation 
findings when It Is on a more stable financial footing. 

• Inter- an d Intra-Organlzatlonal Relations : Turf battles over 
cl lents , staff and other resources can compromise the 
evaluation effort. If for example, cooperative support among 
agencies 1s lacking, the evaluator may find access to Important 
sources of Information curtailed or delayed In ways that 
negatively Impact the evaluation. A comprehensive evaluation 
planning effort will Include strategies to ameliorate or 
compensate for difficult organizational relations. 

Improving Program Evaluability 

Some program attributes may Impinge upon JTPA evaluation planning In 
ways that are difficult and/or costly to remediate right away. For 
example, the data limitations Imposed by a particular HIS may be fairly 
rigid and uncompromising for evaluation plans In the near term. 
However, other limiting factors may be more amenable to change In favor 
of Immediate evaluation needs. 



An evaluability assessment Is not Intended to act as a discouragement 
to evaluation. Part of the assessment task Is to help program 
operators determine what evaluability factors can be manipulated to 
enhance overall evaluability. Once those evaluability factors subject 
to Influence are Identified, evaluation staff can actively work with 
program administration and staff to create a program environment that 
Is more receptive to evaluation. Staff can tackle not only technical 
evaluability factors such as data collection levels or methods, but 
also organizational factors such as program goal definition and 
Interagency communication. Thus , the benefits of evaluability 
assessment extend beyond preparation for useful, feasible evaluations. 
An assessment encourages program examination and Improvements Important 
In their own right, apart from any evaluation activity to follow. 
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TO WHAT EXTENT WILL THE EVALUATION BE UTILIZED? 
As the field of evaluation research develops, there is Increasing 
concern over making such research more Immediately useful to 
practitioners In the field. This concern 1s particularly underscored 
In the context of JTPA where limited administrative funds are available 
for evaluation activities. The feasibility of an evaluation and Its 
usefulness are obviously Intertwined, as the previous discussion on 
evaluablllty implies. The focus In this section is on Increasing the 
usefulness of an evaluation, especially In terms of Increasing the 
chances of its utilization . (Utility and utilization are not the same 
thing: an evaluation's results may be useful , In the abstract, but 
still not used .) The rest of this section looks at barriers to 
utilization of evaluation and presents suggestions for minimizing these 
barriers . 

Barriers to Utilization 

The previous section touched primarily on potential barriers to 
planning, conducting and Interpreting the results of an evaluation. An 
evaluablllty assessment is also Important in uncovering potential 
barriers to utilization , particularly barriers associated with 
organizational features of a program. What are some of ihese 
organizational barriers? While a fuller discussion of this question is 
reserved for the next chapter, the following outline provides a glimpse 
of common barriers to utilization. 



UTILIZATION OF EVALUATION RESULTS: 
SOME POTENTIAL BARRIERS 



• Organizational Inertia and resistance to change. 

• Miscornmunlcation between evaluators, those within the 
program being evaluated, and other potential users of the 
evaluation results. 

• Misunderstandings about the purposes of an evaluation. 

• Lack of organizational Involvement In or commitment to the 
evaluation process. 

• Failure to sufficiently connect the evaluation to other 
planning efforts. 

• Overly lengthy timeframe for accomplishing evaluation. 

• Unresolved tensions or conflicts between different 
organizational levels or branches of a program. 

• Evaluation team perceived as lacking Independence and 
neutrality. 

• Evaluators lack credibility. 

• Evaluation findings not clearly presented or adequately 
disseminated. 



Increasing Evaluation's utility and Utilization 
Many evaluators are now playing a more activist role In ensuring the 
utilization of their findings by program administrators and others. 
Such a role demands that the evaluation group communicate and work more 
In concert with users from the earliest stages of evaluabillty 
assessment to the issuing of a final report. The following ten points 
summarize the kinds of steps an evaluator can take to build utility 
into the evaluation process from the very beginning. 

1 • Identify "stake-holders* and users of the evaluation . 

The term "stake-holder" 9 refers to anyone who has a stake 1n 



The term "stakeholder 11 Is taken from Carol Weiss, "Measuring the 
Likelihood of Influencing Decisions" 1n Leonard Rutman, ed. f 
Eva luation Research Methods: A Basic Guide . (Beverly Hills, 
CA: Sage Publications, 1980. pp. 159-190. 
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the evaluation process and Its results. Stake holders can Include 
program f under s f administrators f planners f pol icy- makers f 
front-Hne staff, clients and client-advocates. While not all of 
these groups may be directly Involved in the evaluation process, 
1 1 1 s important to know who these parties are and how their 
Interests or concerns might affect the evaluation and Its utility. 

2. Involve stake-holders In the planning process . 

Where feasible, stake-holders who are potential users of the 
evaluation results need to be Involved early on In the evaluation 
planning process (starting with eva'luabillty assessment) for at 
least two major reasons . F1 rst , potential users have to be 
committed to the particular evaluation chosen and believe In its 
utility to JTPA program Improvement. Participation In the 
planning process helps to build user understanding of and 
commitment to the evaluation effort. Second , user input helps 
focus the evaluation on the legitimate concerns and Interests of 
the various users. The evaluation is more 1 ikely to produce 
Information that critical actors In the program will want to use, 
as opposed to information that is of Interest only to the 
evaluation staff. 

3. Educate potential users . 

One problem In basing evaluation around user input Is that users 1 
initial focus may be restricted to Issues of Immediate 
programmatic concern. For example, users at the SDA level may 
primarily be concerned with compliance and monitoring issues. 
While these concerns need to be addressed, evaluation planning can 
provide users the opportunity to explore broader evaluation 
options . 

4. Focus evaluation on users 1 key questions . 

Ultimately, the evaluation must yield information that users feel 
will be Important to them in answering questions about JTPA 
programs. Achieving such a focus Is not always easy because 
different users wl 1 1 bring to the planning process di f ferent 
perspectives as to what Information Is most useful and Important 
to obtain. 

5 * Maintain neutrality and impartiality . 

Tc be useful, an evaluation must be credible to JTPA 
decision-makers and others. In large measure, such credibility 
rests with the Independence and neutrality of the evaluation 
staff. Positioning of the evaluation staff within an 
organize ion, and the relationship of that staff to JTPA 
administrators and policy-makers are Important factors influencing 
the perceived or actual independence of that staff. 

6. Develop mechanisms for Interim feedback . 

The evaluation process is often a lengthy one. Where possible, 
Interim reports , newsletters or presentations help sustain users 1 
Interest and commitment to the evaluation. Traditionally, 
evaluators have avoided such Information exchange with users for 
fear that such feedback might contaminate data. However, more 
recently, commentators have suggested that such fears have been 
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overstressed and need to be balanced against the practical 
advantages such Interim feedback offers to practitioners. 

Develop a dissemination strategy . 

Traditionally, dissemination 1s almost an afterthought to an 
evaluation plan, Involving little more than sending copies of the 
final report to the evaluation funders or perhaps seeking 
publication of the findings 1n an academic journal. Expanding the 
usefulness of an evaluation, however, calls for a broader, more 
creative approach to dissemination. Such a broader approach might 
Involve: 

• Targeting important users and other Interested parties 
ahead of time and maintaining contact with these groups via 
newsletter or Interim reports. 

• Planning 1n-person presentations to various users to allow 
for direct questions and answers about evaluation findings. 

a Where appropriate, identifying other opportunities to 
present findings to a larger forum of practitioners, as 
well as researchers, such as a conference or special 
publication. 

• Discussing ahead of time how users might be involved in 
dissemination and whether users will be given formal credit 
or recognition when findings are presented. 

• Considering in what manner public affairs staff might 
assist in presenting findings. 

Produce a clear, well-written report of findings . 
A lengthy, jargon-filled report emphasizing the technical 
aspects of an evaluation creates what Weiss terms "coanitlve 
obstacles" to its utilization. Utility of an evaluation 
obviously Increases 1f findings are pitched to a broad audience 
of interested parties. Ways to Increase readability include: 

• Presenting a separate executive summary of findings which 
highlights the most important conclusions. 

• Placing technical information, where possible, in a 
separate chapter, appendix or in footnotes. 

• Prominently featuring, through formatting and placement, 
the main evaluation questions, interpretations, findings 
and recommendations. 

• Adding a glossary of technical terms, if necessary. 

• Packaging evaluation findings differently for different 
audiences. For example, pairing technical summaries of 
findings relating more to research issues or conclusions 
with more "user friendly" summaries relating to policy 
issues of current Interest to decision-makers. 



9. Present findings In a timely fashion . 

Timing Is all Important In the reporting of evaluation 
findings. If too much time has elapsed between evaluation 
planning and reporting, the Information presented may no longer 
be fresh or relevant to users. On the other hand the user's 
call for timeliness must be judged against the need to acquire 
reliable and valid information through acceptable research 
procedures, all of which takes time. The point is not to rush 
through with dubious results, but to agree upon a responsible 
timeframe Initially and then stick to it. 

1 0 . Imbed evaluation 1n ongoing planning cycles . 

Ideally, evaluation plays an integral role 1n an organization's 
overall planning processes. Evaluation provides feedback at 
critical junctures in a program cycle, allowing planners and 
policy-makers to make Informed judgments about the future 
direction of the program. If evaluation Is simply tacked on to 
JTPA programs as an afterthought and Is not coordinated with 
other JTPA planning efforts, then evaluation's utility is 
likely to be diminished. 

CONCLUSION 

This chapter Is meant to encourage JTPA evaluation planners In the hard 
exploratory planning work that establishes a solid foundation for later 
evaluation activities. Scrutinizing a program for evaluabillty may 
sound like unnecessarily discouraging or time-consuming work. However, 
discovering potential program constraints to evaluation early on will 
give evaluation planners an edge In introducing feasible evaluation 
activities In a more effective manner. 



A related concern Is whether the evaluation findings will be 
sufficiently utilized to justify the evaluation effort. In order to 
ensure ultimate use, planners need to anticipate potential barriers to 
utilization and actively build Into the evaluation plan strategics for 
increasing the usefulness and utilization of findings. Sir.ce the 
organizational context Informs and shapes the evaluation strategy in 
important ways, the following chapter looks more closely *t in* 
relationship between organization and evaluation. 
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CHAPTER 4 

UNDERSTANDING THE ORGANIZATIONAL CONTEXT: 
PART 1: ORGANIZATIONAL ISSUES FOR JTPA 

EVALUATION 



How Will the Organizationei Context impact Evaluation? 

Overcoming Organizational Inertia 
The Evaluation and the Evaluated 
Reducing the Threat of Evaluation 

Who Wiil Participate in and Support Evaluation? 

Who Should Do the Evaluation? 
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CHAPTER 4. 

UNDERSTANDING THE ORGANIZATIONAL CONTEXT: 



PART 1 : ORGANIZATIONAL ISSUES FOR JTPA EVALUATION 



INTRODUCTION 

Evaluation does not occur in a social vacuum. Just as political and 
organizational factors influence JTPA program design and operation, so 
will such factors Influence the nature and scope of evaluation. The 
JTPA organizational context is complex, cross -cutting all levels of 
government and embracing numerous agencies and organizational agendas. 
Because of this complexity, understanding how organizational factors 
might Intervene to help or hinder evaluation Is especially critical to 
the JTPA evaluation planning process . For example, In such a 
multi-layered program as JTPA, various organizational tensions and 
conflicts are bound to occur. The system may not have mechanisms to 
respond, and the conflicts can spill over Into the evaluation. 



In addition, the evaluation Itself may subtly Influence program 
processes and outcomes- Therefore, not only the context In which 
evaluation occurs, but also the manner In which evaluation Is carried 
out (In Interaction with the context) Is also Important. For these 
reasons, preliminary planning for evaluation must Include a focus on a 
third preliminary planning question: How does the organizational 
context Impact evaluation ? When Ignored during the evaluation planning 
stage, underlying organizational conflicts can erupt, creating 
tremendous roadblocks to later Implementation and utilization of 
evaluation. Related questions are: Who will participate In and 
support evaluation? Who should do the evaluation? 



The purpose of this chapter Is to more fully explore those various 
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organizational Issues likely to Impact JTPA evaluation efforts at the 
state level. The first part focuses on the most general Issues 
affecting JTPA evaluation efforts, while the second part concentrates 
on evaluation and state-SDA relations. 

HOW WILL THE ORGANIZATIONAL CONTEXT IMPACT EVALUATION? 
For the evaluation planner, the challenge Is to Identify and 
knowledgeably work with organizational constraints and supports to 
evaluation. Since these constraints and supports will vary from state 
to state, the Intention here Is to provide a general framework for 
Incorporating organizational Issues Into the evaluation plan. 

Overcoming Organizational Inertia 

To accomplish their specified missions, organizations create structures 
to promote stability and efficiency. Organizations develop structures 
which establish chains of authority and accountability, standardize 
operations, and routlnlze and parcel out work In a specific manner. In 
creating stable structures, organizations also create vested Interests; 
a major goal of the organization becomes self-preservation. Over time, 
the very structures developed to enhance the organization's efficient 
functioning have a tendency to become rlgldlfled and resistant to 
change. Change means more uncertainty and , as such, constitutes a 
threat to the organization and Its vested Interests. 

The logic of evaluation, on the other hand, Is based on the potential 
for change . Ideally, evaluation feedback offers a rational mechanism 
for planned change In the Interest of program Improvement. Therefore, 
as a harbinger of such change, the evaluator can expect to encounter 
some natural organ 1 zatlonal resistance to evaluation activities . 
Sometimes the resistance Is not active, but takes the form of passive 
Inability to mobilize for an evaluation effort. Sheer organizational 
Inertia- the urge to follow time-honored structure and patterns which 
have shaped the organ 1 zat Ion 1 s Identity- Inhibits the evaluation 
undertaking. 

Some might suggest that since JTPA 1s relatively fresh legislation, Its 
programs have not yet had the time to solidify and build up an 
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organizational Inertia. However, while JTPA legislation 1s new, 1n 
many cases the program structures and personnel utilized to Implement 
1t are not. Program continuity from CETA days has undoubtedly helped 
many states and SDAs to mobilize for a JTPA effort more effectively. 
By the same token, program continuity means that many JTPA 
organizations are actually long-established with well-defined Interests 
and are likely to resist evaluation geared towards program change. On 
the other hand, In an age of shrinking public resources, JTPA and other 
programs are under constant external pressure to Improve (I.e., be more 
productive with fewer resources). Evaluation provides a tool for such 
change which need not threaten the security and continuity of the 
organization. 



Overcoming organizational Inertia or outright resistance to evaluation 
may present more of a challenge than the actual evaluation Itself. 
JTPA's complex administrative structure may demand that not one, but 
several separate organizational entitles be mobilized to cooperate and 
participate In evaluation activities, If those activities are to be 
meaningful . 



To accomplish this mobilization, evaluation planners may have to 
broaden their traditional role to Include education, mediation, 
communication, and public relations activities preparatory to planning 
the evaluation Itself. A common organizational fear Is that the 
evaluation results will only point out program weaknesses and damage 
program credibility. Program administrators need to be assured that 
the evaluation results can enhance program credibility In several 
ways: The fact that a program embraces evaluation as a tool for 
Innovation and Improvement Itself sends a positive message to program 
sponsors. Moreover, a balanced program evaluation will help Identify 
program strengths, as well as weaknesses, uncovering program 
accomplishments which compliance measures do not take Into account. 
And finally, evaluation may produce Information that compensates for or 
explains lower compliance with performance standards. 

The Evaluator and the Evaluated 

Even if only temporarily, the evaluator also becomes a part of the 
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organizational landscape 1n which he or she 1s operating. How those 
being evaluated perceive the evaluator and how the evaluator, In turn, 
interacts with those he or she observes, must Inescapably Influence the 
evaluation process. For these reasons, the evaluator must be sensitive 
to his or her role as an Innovator within the organization and 
anticipate potential difficulties arising from that role. The first 
big challenge for the evaluator Is to reduce the threatening aspects of 
this role. 

Regardless of the specific purpose behind an evaluation, the evaluator 
wishes to be regarded as a facilitator of positive change within the 
system being evaluated. However, It is difficult for those being 
evaluated to embrace the evaluator 1 s most positive point of view: 
their natural prejudice is that the evaluator has come to point a 
disapproving finger at what they are doing wrong. If nothing is done 
to soften this negative predisposition to the evaluator, if no 
assurances and protection are given to the evaluated, then an 
evaluator's presence is likely to Induce a defensive posture. 

For example, in one case, JTPA evaluators were Investigating the 
Impacts of a special state program through use of a comparison group of 
non-participants. When the evaluation was in progress, the evaluators 
discovered that program staff, In their eagerness to prove the 
program's worth, became unofficial program gatekeepers- assigning for 
JTPA services only the most obviously job-ready. As a result, It 
became difficult to assess whether positive outcomes were due to the 
program services or to the select nature of clients receiving those 
services. 

The evaluator unavoidably has an Impact not only on the social climate 
of a program (an Intruder on sacred soil) but also on the working 
conditions within the program. In requiring Interviews and planning 
meetings, the evaluator distracts staff and administrators from their 
regular work load. Whether staff perceive evaluation duties as a 
burden or an intrusion depends, in part, on the sensitivity of the 
evaluator and how well staff are briefed as to the nature of the 
evaluation and the importance of their role in the evaluation process. 
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In a positive context, evaluation Interviews and planning meetings can 
offer staff a chance to be heard and make a meaningful contribution. 
In addition, an evaluation project generates Its own phoning, typing 
and other office requirements, which may place extra burdens on an 
already overloaded support staff. Resentments over this new work can 
build If expectations for program staff participation are not Initially 
clarified with the evaluation staff. 

The evaluator's (or evaluation staff's) perceived status may also be 
significant to the success of the evaluation. If, for example, the 
evaluator 1s perceived as too closely aligned with the administrative 
power structure, this perception may Impair the credibility of the 
evaluator and his or her ability to carry out evaluation functions. On 
the other hand, If the evaluator 1s perceived as lacking sufficient 
administrative support, he or she may be seen as "marginal" In relation 
to ongoing program operations. The message Is that evaluation Is not 
really valued and participant cooperation In the effort may be 
undermined. 

Finally, the evaluator must confront the possibility that his or her 
presence constitutes an additional Intervention, or Independent 
Influence which may affect the program 1n an unknown fashion. If, for 
example, the evaluator Is seen as a threatening presence, staff morale 
and program effectiveness may decline. Alternatively, staff may take 
extraordinary measures which artificially and temporarily boost program 
performance. Even If the evaluator Is viewed In a strictly neutral 
light, the subjects of the evaluation (who may range from JTPA clients 
to JTPA policy-makers) may simply react to the process of being studied 
(the well-known Hawthorne effect). Evaluation activities such as 
surveys and Interviews may themselves constitute a contaminating 
Influence on the evaluation results. 

While such Influences cannot be totally eliminated, evaluators can seek 
to minimize their Impact on the research process. The sample checklist 
which follows summarizes how the evaluator's role can be clarified, not 
only to help the evaluator but also the program staff, administrators 
and evaluation sponsors who must Interact with the evaluator. 
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CLARIFYING THE EVALUATOR 1 S ROLE 



YES NO 

I i I I 1- Evaluator-Admlnlstrator Authority Is there 

written agreement about the evaluation 
decisions and who will be Involved In making 
those decisions? 

I I I I 2. Evaluator-Admlnlstrator Responsibilities Have 

responsibilities for both administrators and 
evaluators been clearly defined In writing? 

I_l iZj 3. E valuator-Staff Responsibilities Has the 

degree of program staff participation and work 
responsibilities been defined and put In 
writing? 

I_! I I Communications Have formal channels of 

communication among the various evaluation 
participants been established? 

!_l l_l 5. Resources Utilization Are there written 

agreements about the use of In-house resources 
(e.g. f phones , copying equipment , office 
space, etc.) by the evaluation staff? 

I I I I 6. Disagreements Are there written procedures 

for resolving disagreements between program 
and evaluation staff when they arise? 

I I l_l 7. Briefing Staf f L^ve program staff been 

briefed on the above relevant agreements? 

I I I I 8. Involving Staff Are opportunities for 

Interaction and exchange of Information with 
program staff scheduled Into the evaluation 
process? 

I ! I ! 9. Introducing the Evaluators and the Evaluation 

Has Initial time been set aside for Intro- 
ducing the e valuator and evaluation plans to 
the staff and allowing for staff questions. 

I I I I 10. Evaluator Influence on the Program Have the 

evaluator 1 s planned activities been assessed 
for possible Influence on program operations 
and outcomes? 



Adapted from: Kay Adams and Jerry Walker, Improving the 
Accountability of Career Education Programs: Evaluation Outlines 
and Checklists , the National Center tor Research 1n Vocational 
Education, Columbus, OH, 1979 
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Reducing the Threat of Evaluation 

The evaluator 1s not automatically doomed to alien status within a 
hostile and mistrustful program environment. Although some 
organizational factors may be beyond the evaluator 1 s control f the 
evaluation plan can Include several strategies to demystify evaluation 
and reduce a program staff's Initial fears about the evaluator and the 
evaluation process: 

• Involve not only program administration, but program staff, as 
well , In the Initial and subsequent evaluation planning 
activities In order to enhance user understanding and commitment 
to the evaluation. 

• Make clear to program personnel the purposes and anticipated 
consequences of the evaluation. Ideally, consequences :enter 
around constructive program change so that the program ran be 
allowed room to fall, but then move on. Remove the threat of 
sanctions being attached to the evaluation to the degree 
possible. 

• Emphasize the evaluation of programs , not personnel . The more 
emphasis placed on evaluating the program attributes, as opposed 
to staff attributes, the less threatening the evaluation 
process. If staff Inadequacies are a central concern, then 
other vehicles besides program evaluation should be considered 
to address this concern. 

• Establish clear lines of authority separating evaluation staff 
from program administration staff. 

• Introduce an Initial evaluation effort Into the least 
threatening program situation. For example, some states have 
Initially focused JTPA evaluation on programs outside mainstream 
Title II, where programs are not subject to the same performance 
standards pressures. Such a strategy allows a state to develop 
a track record of purposeful evaluation before tackling 
mainstream programs. 

• Assure confidentiality to clients, staff and all other 
participants In evaluation. 

• Select evaluators whose organizational status Is perceived as 
most neutral and non-threatening. 

WHO WILL PARTICIPATE IN AND SUPPORT EVALUATION EFFORTS? 
Numerous distinct state and local level organizations are Involved In 
the administration, planning and Implementation of JTPA activities. At 
a minimum, the state's employment security agency, vocational education 
groups, welfare offices and the State Job Training Coordinating Council 
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(SJTCC) are all participating at, the state level. In some states, the 
governor's office, the legislature, economic development and other 
related agencies alsc play an active or Influential role. In addition, 
depending on the nature of the evaluation, SDAs, PICs and local 
government will also bo Involved. All theie organl zatione i actors have 
developed a stake within the JTPA system and therefore hav^ a 
legitimate Interest *n eva luatlon design, Implementation end outcomes 
which affect them. 

Before launching Into a full-blown evaluation effort, c.ie should 
consider the roles these various organizational acto r s play within JTPA 
and how supportive of evaluation they are likely to be. How active or 
central a role does each organization play? How receptive to or 
constrained bi evaluation are key actors? What explicit or "upllcit 
agency agendas might affect the evaluation effort? Ignoring the 
Interests of a particular JTPA stake-holder in the planning phase may 
Impede the evaluation In later implementation and utilization phases. 
(The second part of the chapter focuses explicitly on the SDA as an 
Important stake-holder.) 

The same Issues apply to subdivisions within a single JTPA agency. For 
example, a lead state agency may house JTPA planning staff in one 
division, technical services In another, and research and analysis in a 
third. Each of these staffs may be relatively Insulated from each 
other, maintaining a separate organizational identity and set of 
concerns. Yet each may have an important role to play In evaluation In 
terms of advising evaluation staff on technical matters, assisting in 
the evaluation planning or actually implementing parts of the 
evaluation. Clearly, agency or divisional support from the top is a 
necessary prerequisite to actual Implementation of evaluation. 

Evaluation activities that cross divisional boundaries, while providing 
extra challenges to planning and coordination may also provide unique 
opportunities for the exchange of information and ideas within the 
organization. Since evaluation often requires special coordination 
among different units, the evaluation process can create a supportive 
context for interaction across territorial lines. Such Interaction can 
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Itself be valuable In Informing people about decision-making and work 
agendas In different divisions, reducing divisional Isolation and 
Improving coordination of resources. 

Interagency Relations: Conflict or Cooperation? 
It Is not sufficient to know who the organizational actors are and what 
their stakes In JTPA entail; one also needs to know how these various 
groups Interact with one another. Existing organizational patterns of 
Interaction are often best understood In a historical context. Some 
organizational elements of JTPA (like the State Council) are totally 
new, while others have an Important history predating JTPA and 
Influencing current patterns of Interagency cooperation, communication 
and conflict. 

Do agency heads regularly communicate with each other? Are there 
unresolved turf battles over JTPA or other program areas? Have 
personality conflicts marred Interagency cooperation In the past? 
These are the kinds of questions an evaluation planning group will have 
to pose and answer In order to lay the organizational groundwork to 
support an evaluation effort. 

Conflicting Interests 

Sometimes organizational Interests are pitted against each other In 
ways that make coordinated evaluation very difficult. Conflicting 
Interests are most likely to arise where two agencies share the same 
client base, as Is the case with many JTPA and welfare programs. 
Competition between these two programs can be particularly Intense when 
the fuller funding of JTPA has translated Into less funding for welfare 
recipients enrolled In JTPA, and It 1s no longer In the Interest of 
welfare agencies to refer clients to JTPA. Nor, for that matter, Is It 
In their Interest to participate In an evaluation which might validate 
JTPA at welfare's expense. 

If agencies have a history of poor communication or turf battles over 
who should administer what programs or who should set policy, this 
history can spVH over Into and stymie evaluation efforts 1n Important 
ways : 
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• Access to necessary data, program documents or clients may be 
delayed or made more difficult. 

• Otherwise useful In-house resources may not be discovered and 
shared. 

• The organizational Input necessary for formulating useful 
evaluation questions may not occur. 

• The general utility of evaluation findings may not be recognized 
by Important decision-makers. 

These kinds of potential obstacles are especial?/ worth considering If 
a process evaluation Is contemplated. Access to various agencies and 
rapport with agency staff will be Important to the evaluator hoping to 
get at key processes and Interactions relevant to the JTPA system. 

Conversely, Identifying potentially positive Interagency connections 
provides a base on which an evaluation effort can be built. Whatever 
the organizational configuration, the planning role cannot remain 
purely technical. The evaluation planner may need to play Information 
broker and mediator, acting as a conduit to open up or enlarge channels 
of communication and cooperation. 

Cooperative Evaluation Planning 

If JTPA evaluation Is not to be lost In a thicket of organizational 
agendas, It Is Important that central actors are able to jointly 
participate In planning efforts and arrive at some common understanding 
as to how evaluation Is to benefit JTPA as a whole. Building up such 
multi-faceted participation Is a challenging task because JTPA concerns 
multiple actors with multiple Interests, needs, and fears, who are 
often not used to working across divisional boundaries. 
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As stake-holders In the JTPA system, agencies/actors need to feel that 
they are each getting something out of participating 1n the evaluation 
effort. A crucial task becomes eliciting from primary actors what It 
Is they are willing to give and get in return as participants In the 
evaluation process. Also the task 1s to help sensitize actors to each 
other 1 s concerns , bringing covert Issues Into the bargaining arena 
(e.g., the perennial problem of data acquisition across agencies) so 
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that necessary agreements can be negotiated upfront before evaluation 
commences. 

WHO SHOULD DO THE EVALUATION? 

The organizational context should also Influence who plans, Implements 
and administers a JTPA evaluation. Should the lead agency have primary 
responsibility for evaluation or should a policy-making body like the 
State Council? Or should an organization more removed from the JTPA 
system have primary evaluation responsibilities? Should evaluation 
responsibilities be divided? Clearly, g ; ;ven enormous organizational 
variation across states, no one organization Is the "right" place to 
house an evaluation effort. What works well In one state setting may 
not be transferable to other states. Below are some factors In 
choosing and locating an evaluation staff. 

Authority Structure : The position of an evaluation staff 
within an organizational hierarchy Is important. Ideally, evaluation 
staff will be sufficiently detached from the existing hierarchy so that 
they are not perceived to hold any direct power over those being 
evaluated or, conversely, those In a program being evaluated do not 
have direct authority or Influence over the evaluators. Such 
detachment Is often sought by contracting out to a private consultant 
or establishing an Independent evaluation unit. 

When the head of an evaluation unit reports directly to chief 
decision-makers In an organization, evaluation activities are more 
likely to be better supported (fiscally and politically) and evaluation 
Information better utilized by managers and policy-makers. Such a 
direct link to power holders, however, may need to be offset with extra 
efforts to bring a range of appropriate division administrators and 
relevant staff Into the planning process. Otherwise, there Is the 
danger that those lower down will feel compromised by or excluded from 
Important decision-making and become less supportive of the evaluation 
effort. 

When an agency 1s attempting Its own 1n-house evaluation, sufficient 
detachment of evaluation staff may be more difficult to achieve. This 
1s not to argue that self-evaluation should be abandoned. Rather, the 
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financial and other practical merits of this approach need to be 
weighed against the potential structural drawbacks of having a less 
organizationally autonomous evaluation staff. 

If an In-house evaluation unit 1s used, a key Issue Is placement of 
that unit. When the unit Is not completely separate from other 
operations, Its members may be In the uncomfortable position of 
evaluating JTPA operations managed by people above them In the 
organizational hierarchy. 

Compliance vs. Evaluation : The JTPA authority structure at 
the state level Is partially defined by who conducts compliance-related 
activities. Many states have developed special monitoring and 
compliance units which routinely collect and analyze JTPA program data 
and audit certain aspects of JTPA program operations. Since these 
units are already collecting some Information about JTPA and since 
evaluation Is often viewed as an elegant offshoot of monitoring, the 
temptation is to lump evaluation activities In with ongoing monitoring 
and compliance operations. (This tendency Is probably also reinforced 
by the CE1A legacy of mingling compliance and technical functions under 
one roof.) 

From a purely technical standpoint, piggybacking evaluation onto 
ongoing monitoring operations may make sense: staff are familiar with 
the data and with program operations and personnel. However, from an 
organizational standpoint, such an arrangement may be quite 
problematic. As mentioned earlier, downplaying the threatening aspects 
of evaluation and enlisting the cooperation of those being evaluated Is 
an important Ingredient to planning a successful evaluation. The 
neutral, non-threatening posture an evaluation staff seeks Is readily 
compromised in the eyes of those being evaluated if that same staff is 
also connected with compliance activities. The Inherently threatening 
aspects of evaluation are heightened by the fact that the office which 
evaluates is also the office which critiques and sanctions. A 
compromise approach might be to Involve monitoring and compliance staff 
as special evaluation consultants who can provide unique Information 
and insights into JTPA program operations, while other staff actually 
Implement the evaluation. 
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Independence an d Neutrality : An evaluation staff's perceived 
neutrality Is closely connected to Its position 1n the organizational 
hierarchy. If the objectivity of evaluators Is questioned either by 
decislon-makers or those being evaluated, the whole purpose of the 
evaluation effort may be called Into question and the potential utility 
of that effort lost. 



The quest for neutrality does not Inevitably lead to expensive outside 
consultants. First, hiring outside consultants does not automatically 
remove the suspicion of b1as--outs1de evaluators may merely be viewed 
as an extension of those who hire them. Second, there are alternative 
approaches to state JTPA evaluation that sufficiently meet the 
requirements of Independence and neutrality. For example, evaluation 
can be accomplished through an Independent research unit under the 
State Council, under the lead agency or under another state authority 
outside JTPA. (For a further comparison of different evaluation 
staffing strategies, see Chapter 10.) 

Trust: Trust Is another Important consideration In deciding who 
Is best able to carry out an evaluation effort. Trust enhances the 
ability of the evaluator to gain entry to a program and elicit 
Information and assistance from program administration and staff. The 
fact that the relationship between SDAs and state JTPA offices In many 
states has been characterized by a certain amount of mistrust 1s 
therefore significant to the evaluation undertaking. 

An evaluator 1 s neutrality does not necessarily guarantee trust or 
vice-versa. In fact, trust may be based on the evaluator's perceived 
Positive bias towards a program. In choosing the evaluation staff, 
trade-offs may have to be made between who has greatest rapport and 
access to program Information and who has greatest neutrality and 
Independence. 



Competency: Technical competency of an evaluation staff Is a 

primary consideration to factor into a decision about how to build an 

evaluation capability. Without proper technical expertise, an 

evaluation 1s more likely to waste resources and produce results of 




questionable validity and usefulness. However, technical competency 
and efficiency, while of primary Importance, should not be the sole 
criteria for location of an evaluation effort. In addition to 
traditional notions of competency and expertise, familiarity with JTPA 
programs and the ability to maneuver within the system and get things 
done are also Important attributes for an evaluation staff. 

Coordination Capabilities : The more comprehensive the 
evaluation effort, the greater the need to Involve and coordinate. Who 
1s best able to perform vital coordination efforts--to bring Interested 
parties together In critical planning stages, to establish Interagency 
agreements about data and resource sharing, to bridge communication 
gaps when necessary? Here again, some argue that these critical 
non-technical competencies must be obtained by hiring an outside 
consultant whose vision can transcend the narrower perspectives of 
Individual JTPA personnel. On the other hand, in-house staff, by 
virtue of their superior knowledge of Interagency history and 
personnel, may also be In a good position to perform such coordination 
functions. 

CONCLUSION: DEVELOPING THE ORGANIZATIONAL 
CLIMATE TO SUPPORT EVALUATION 

This chapter has examined some of the major organizational Issues 
confronting a JTPA evaluation planning effort. In every stage of the 
evaluation process, organizational factors can exert profound 
influences on that process. If organizational support 1s lacking, the 
evaluation effort may flounder and ultimately fall . Traditional 
evaluation planning begins with the assumption that the organizational 
context Is set. In contrast, the assumption presented here Is that 
JTPA evaluation planning must actively consider the organizational 
environment In which that planning takes place. Planning expands to 
include not only a preliminary organizational assessment of those 
factors likely to influence evaluation, but also preliminary strategies 
for building better organizational support for evaluation. Some 
suggested strategies, implicit in much of the preceding discussion are 
encapsulated as follows: 
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Develop leadership support; The Interest and cooperation of 
agency heads and other key administrators 1s Important to obtain 
before planning reaches too advanced a stage. 

Educate key decisi on-makers and their staffs : Decision-makers 
are often unaware of the benefits of evaluation and need first 
to be educated before they will support evaluation. Educational 
efforts might Include circulating policy papers, promoting 
conference attendance, or sharing the results of eve.luatlon 
activities In other states. Essentially, leadership needs to be 
convinced that supporting evaluation, even though results might 
be less than positive, Is a politically responsible position to 
take. 

Seek legl s latl ve support : Some states have built evaluation 
Into JTPA partly through legislative requirements and supports. 
As a result, evaluation may be perceived as a more valued 
activity, 1n which It Is Important to participate. 

Involve k ey actors : Preliminary meetings with key actors In 
the evaluation process will help shape an evaluation approach 
that accommodates a variety of concerns and does not exacerbate 
Inter- or Intra-agency conflict. Staff as well as 
administrators need to be Included In early planning and/or 
briefing meetings. 

Identify side-be nefits of evaluation for different 
participants: In addition to the desired Information 
evaluation Is expected to provide, users will want to know about 
particular (often unanticipated) side-benefits evaluation might 
yield. Of tan these side- benefits are Intangible such as 
Improved ctjency coordination or tnore positive Interagency 
relations. (For more on this theme refer back to Chapter 1 
and to thft discussion on measuring evaluation benefits In 
Chapter 7.) 
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Develop a dvisory groups : To ensure greater understanding of 
and commitment to evaluation, some states sponsor evaluation 
advisory groups. Group members not only may Include agency 
representatives, but outside professionals or other citizens to 
lend additional support and credibility to the endeavor. 

Develop i nnovative funding and staffing alternatives : Sources 
of support for evaluation exist beyond the usual organizational 
channels. Moving outside an agency for evaluation resources can 
extend the base of Interest and support for such activity. 
(More on this point In Chapters 9 and 10.) 

£yl interagenc y agreements and assurances In writing : 

Successful evaluation often depends upon Interagency cooperation 
and sharing of resources. Since control of resources Is always 
a sensitive organizational issue, negotiated agreements about 
access to data, clients, staff and other resources need to be in 
writing to avoid future misunderstanding. 

Use a te am planning approach : A team approach to planning 
makes sense where a lot of Inter- or intra-agency coordination 
and communication Is necessary for accomplishing evaluation 
tasks. Even if an outside evaluator Is brought in to do the 
work, a team might also play a useful advisory role, providing 
a mechanism for more direct organizational involvement and 
commitment to the evaluation. 
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CHAPTER 4. 

UNDERSTANDING THE ORGANIZATIONAL CONTEXT: 



PART 2: STATE EVALUATION AftD THE SDAs 

INTRODUCTION 

c.o far we hav<! discussed organizational Issues affecting evaluation 
primarily at the state level. Many state evaluation activities will 
also Involve, concern, Influence, or directly Impact SDAs. 
Concurrently, SDAs will have their own organizational agendas with 
which state evaluation activities Ideally should mesh. The factors 
which affect SDA participation, cooperation or support In state- 
sponsored evaluation of JTPA are the subject of this following section. 

HOW WILL STATE-SDA RELATIONS IMPACT EVALUATION? 
Just as Interagency relations permeate state-level evaluation 
activities, so too do SDA-state relations Impact the overall climate 
for evaluation. Where relations are marked by turf battles, poor 
communication, or mistrust, the purposes of evaluation may be lost. 
Potentially useful evaluation findings may have little Impact on 
program operations because SDAs are not receptive to the Information. 

Recent survey data suggests that for many states, relations with SDAs 
are less than totally cooperative and tranquil. 10 Evaluation 
activities In these cases may be particularly vulnerable to 
organizational cross-fire. If evaluation planners are sensitive to the 



In the early stages of JTPA Implementation, many SDAs apparently 

perceived the state to be simply replacing the federal authority 

In regulating SDAs and limiting their autonomy. See Robert Cook 

et al -» State Level Implementation of the Job Training Partnership 
Act (Rockvllle, HD: Westat, 1984), Chapter 5. 
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causes of state-SDA tensions and develop strategies for reducing them, 
evaluation Itself can serve as a vehicle for Improved state-SDA 
communication and understanding. 

Evaluation and the state Oversight Role 

The role of coordinator and overseer of statewide employment and 
training programs 1s a new and evolving one for most states. The role 
Involves balancing state administrative requirements and 
responsibilities of JTPA against the local Initiative and autonomy also 
cal led for In the legislation. Where this delicate balance between 
state and local Interests lies Is not clearly specified In the 
legislation and not always readily agreed upon by the concerned 
parties. As a result, state Initiated evaluation can be perceived by 
SDAs to be an unwarranted Intrusion Into SDA program affairs. There Is 
the persistent worry that states will use evaluation Information to 
Influence local JTPA operations and undercut local autonomy. 

Even If the evaluation focuses on statewide outcomes rather than 
Individual SDA performance, the SDA may feel uncomfortable If they have 
to contribute time and resources to an evaluation effort from which 
they are not sure what, If any, benefit they will derive. In one 
statewide evaluation effort, an SDA participant complained that the 
state sent reams of data printout to them without any accompanying 
analysis or explanation as to what had been done to their raw data or 
how the SDA Itself might profitably use this evaluation Information. 

Statewide evaluation policy needs to be based on strategies which meet 
state oversight responsibilities yet also affirm SDA autonomy and 
encourage local level evaluation as well. SDA fears and concerns for 
autonomy may be addressed principally by including SDA decision-makers 
and key staff in the aspects of evaluation planning which most affect 
them. SDA participants need to have a clear understanding of the 
Intended uses of the evaluation and, most Importantly how they will be 
affected by and able themselves to use the findings. In order to 
better work with SD£s, some states have formed SDA user advisory groups 
and have encouraged these groups 1 active participation in evaluation 
planning. Others have found utilizing an independent consultant to 
collect and analyze data and report findings to be helpful 1n reducing 



SDAs' concerns over state encroachment on their territory. 



Divergent Interests in Evaluation 

Because the state and SDAs have fairly distinct administrative roles 
within J1PA, their Interests In evaluation are often widely divergent 
(although not necessarily antithetical). As a result, states and SDAs 
will embrace different evaluation goals, timetables, and approaches. 

As overseer of the entire JTPA system, state JTPA decision-makers have 
a legitimate concern In evaluation questions of a broad, overarching 
nature. One of the primary purposes of evaluation, from this 
perspective, Is to Inform state policy; as a result, state goals and 
timeframes for evaluation may be more complex and long-term. SDAs on 
the other hand, being closer to the front lines of JTPA service 
delivery, have more Immediate concerns relating to managing their 
programs well and meeting performance standards. They are more likely 
to be Interested In evaluation which provides quick turnaround time on 
results that will more Immediately help them to Improve program 
operations and meet performance standards. 11 

Because of these differing concerns, SDAs are unlikely to have an 
Initial enthusiasm for or commitment to state JTPA evaluation efforts. 
Yet, because of the relatively decentralized nature of JTPA, many 
states have to work with SDAs In order to obtain HIS data and other 
kinds of Information they need for evaluation. Reliance on SDA 
cooperation to accomplish state JTPA evaluation goals places states In 
the uncomfortable position of requiring cooperation from an entity that 
has little stake In these goals, unless the evaluation allows for an 
exchange of useful information between the two program levels . 

While this tension between differing evaluation goals Is somewhat 
Inherent to the JTPA system, some states have lessened the tension by 



Fr?n JJPA^ialu_ati on Issues, Priorities and Contingencies at the 

Sk£_- level ; A Report on a National/State Survey of Local 

Constituencies, by Snedeker Scientific and the Seattle-King County 
Private Industry Council for the JTPA Evaluation Design Project 
(Olympla, WA: Washington State Employ »nt Security Department. 
April, 19£t). pp. 27-29. 
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directly Incorporating SOA evaluation concerns Into state evaluation 
efforts. Some have brought SDA user groups Into the evaluation 
planning process, providing SOAs with the specific opportunity to 
clarify the kinds of evaluation Information that would be useful at the 
local level. One JTPA state agency, for example, sponsored a retreat 
with pic and SDA officials 1n order to develop a state evaluation 
agenda that would Include local evaluation priorities. 

The influence of Performance Standards 

The SDA orientation towards program evaluation Is Inescapably 
Influenced by performance standards enforced at the state and federal 
levels. Undoubtedly, for many SDAs the direct and dally pressures of 
meeting these standards lead them to view comprehensive evaluation as a 
luxury for which they lack the time or resources to Indulge In. 

According to recent national assessments of JTPA, performance 
standards, In combination with Inadequate funding, create a climate of 
conservatism at the local level, affecting who Is served and how they 
are served. Interjecting evaluation Into this climate may be perceived 
as too risky a proposition, suggesting unwelcome change. 

Both policy and programmatic change soak up scarce resources with no 
guarantee of Improved efficiency and performance. Even the smallest 
changes, such as redoing an Intake form, Involve significant costs 
which the SDA alone must often bear. In short, evaluation may require 
change, and change may be too risky for an SDA under relentless 
performance pressure to contemplate on Its own without some technical 
assistance or support from the state. Some of the ways states might 
encourage SDA evaluation Initiatives are developed at the end of this 
chapter. 
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Data Sharing Concerns 

A pivotal concern for many SDAs Is the nature and scope of data they 
will be required to provide In a state evaluation effort, rirst, data 
collection may require extra time and effort. Who will absorb data 
collection costs? If the SDA Is to absorb some of the costs, how will 
It benefit? How useful will the data be, from an SDA perspective? 
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Will the data help the SDA answer questions of local Importance? 

Second, how will the data be used? SDAs are understandably concerned 
about the degree to which findings will be connected with sanctions or 
pressures by the state to make programmatic changes. SDAs will also be 
concerned about how findings will be released, and how the public ,iay 
Interpret them. W111 reports be released to the legislature or to the 
press? How much control will SDAs have over presentation of final 
results? 

Finally, some SDAs have expressed concerns that data might be used In 
punitive ways against clients by state and federal governments. Lhould 
automated J1PA data bases be used to ferret out draft evaders, welfare 
abusers, Illegal aliens? Will providing such Information Impair the 
service provider's credibility with disadvantaged clients. 

SDAs concerned about these Issues over data sharing may want specific 
assurances from states dbout ^ow data Is to be colle-'tea and used 
before they can agree to participate In an evaluation effort. In any 
case, It Is a good Idea for state staff and SDA heads to put In writing 
whatever assurances and agreements the organizations make to avoid 
potential confusion and misunderstanding later. On the positive side, 
this kind of joint planning and negotiation over data use can Isad to 
greater mutual understanding of each organl zatlo.- ' s concerns and pave 
the way towards better cooperation and trust. 

HOW CAN THE STATE FOSTER SDA-LEVEL EVALUATION? 
Some states have ventured Into evaluation of specific SDA programs and 
practices; others feel that this territory 1s best left to the 
Individual SDAs. Even If states are not Involved In evaluating 
Individual SDAs, they may play an active role In promoting and 
encouraging local level evaluation. With Its larger pool of research 
expertise and other centralized resources, the state Is In a good 
position to offer a range of evaluation-related services to SDAs 
calculated to stimulate Interest and expand evaluation capabilities. 
Some states are offering or planning to offer special evaluation 
services or Inducements to SDAs. Various evaluation-related services a 
state might make available to SDAs are listed below. 
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• State sponsored TA workshops on evaluation Issues. 

• State clearinghouse on evaluation materials or other resources. 

• Newsletter or briefing sheet on evaluation Issues or findings 
distributed to PICs and SDAs, 

• State sponsored technical assistance team to act as evaluation 
consultants to SDAs. 

• Small technical grants to SDAs fot special evaluation funding 
needs . 

• Partial state funding of demonstration evaluation projects by 
select SDAs to serve as potential models for future evaluation 
activities . 

A recent survey suggests that although state-SDA relations are often 

subject to tensions, SDAs are most satisfied with and receptive to 

1 2 

state technical assistance. The larger, more urban SDAs, however, 
may wish to utilize their own In-house resources and consultants, 
building up their own evaluation capabilities, rather than depending on 
the state for special technical assistance. 



Developing state technical assistance capabilities that will truly be 
helpful to SDAs In planning and Implementing evaluations requires 
preliminary and ongoing exchange with SDAs. Obviously, the Interests, 
needs and capabilities of SDAs will play a large part In determining 
the state technical assistance role. 

CONCLUSION 

State JTPA evaluation planning must Inevitably reach out to the SDAs, 
who, directly or Indirectly, will be Impacted by the evaluation process 
and Its results. SDA administrators have their own program agendas and 
concerns distinct from state JTPA concerns. The SDA 1 s view of 
evaluation Is shaped by their proximity to program operations and the 
accountability and compliance demands placed on them. When the needs 
and sensitivities of the SDAs are honored, evaluation becomes and 
a-enue for enhancing cooperation and communication between state and 
local levels . Ultimately, JTPA programs and those receiving program 
services tan only benefit. 
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SECTION 3 
TOWARDS A JTPA EVALUATION PLAN 



These next chapters continue with an exploration of the evaluation planning 
process. As in the previous chapters, this process is approached through a series 
of key questions confronting the JTPA evaluation planner. In the course of 
answering these questions, the planner follows a roughly sequential set of steps 
culminating in a practical, comprehensive plan for carrying out a JTPA evalua- 
tion effort. 
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CHAPTER 5 

FORMULATING EVALUATION QUESTIONS AND A 
RESEARCH DESIGN 

What Are the Important Questions to Be Answered by Evaluation? 

What Evaluation Approach Makes Sense? 

What Data, Data Collection and Data Analysis Methods Will Be 

Required? 
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CHAPTER 5. 



FORMULATING EVALUATION QUESTIONS 
AND A RESEARCH DESIGN 



INTRODUCTION 

We began the previous chapter with a set of key planning questions 
about program evaluablUty . utilization of evaluation , and the 
organizational context In which evaluation occurs. We now turn to an 
additional set of planning questions which help to define the nature 
and scope of particular JTPA evaluation activities. These questions 
suggest a general planning sequence culminating In a sped f 1c 
evaluation research design. This sequence Is built around the 
following explorations: 

• What are the Important questions to be answered by evaluation? 

• What evaluation approach makes sense? 

• What data, data collection and data analysis methods will be 
required? 

This chapter Is devoted to examining each of these planning 
questions In turn. 

WHAT ARE THE IMPORTANT QUESTIONS TO BE ANSWERED BY 
EVALUATION? 

As has been stressed In the previous section, an evaluation s 
usefulness hinges In large measure on Its providing Information that 
users need 1n order to make more Informed decisions about JTPA 
programs. The actual design of an evaluation, therefore, develops 
around a key, set of research questions aboyt JTPA's e-He ctl v* ;ess , 
efficiency » or program costs . These key questions will, of course, 
vary at different points 1n time across different state and ^ocal 
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program settings, but 1n general, evaluation will concern one or 
more of the following generic questions: 



TYPICAL PROGRAM EVALUATION QUESTIONS 

• Did the program achieve Its stated goals? 

• Did the program have unintended results 
(good or bad)? 

• Was the program Implemented as planned? 

• How might Implementation be Improved? 

• Who benefited most or least from the 
program? 

• Did program participants as a whole benefit 
significantly? 

• What did the program cost? 

• Which program activities were most/least 
cost effective? 



Def'Mng what are the most significant questions to be answered about 
JTPA will help set k e parameters of an evaluation effort early on In 
the planning process. 

Developing E aluation Questions 

Ideally, evaluation questions are generated by the potential users of 
the evaluaM;- (also referred to as "stake-holders"). Users are most 
often pre- administrators, policy-makers or special funders of a 
program; -.ers '.an also broadly Include other stake-holders such 

as staff ri the Inter?-, ted public. As mentioned previously, user 
parti* ioatlon can he crucial In evaluation planning: user Input not 
only in ^ases the user's commitment to the evaluation effort, out also 
focuses that effort on relevant Issues. 

During the question formulation stage, however, evaluation staff do not 
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have to abdicate to users entirely. Sometimes uncovering specific 
questions Is a difficult process; users may have problems developing 
researchable Inquiries about the program. Because JTPA Is so 
tremendously "performance driven" , users may have difficulty moving 
from a compliance and monitoring mode to broader Inquiries. In such 
casss, the evaluation staff can play an Important educative role In 
eliciting or reformulating questions from various users. Ultlrwtely, 
however, user Interests have to be central to the evaluation If the 
findings and recommendations are to have an appreciative audience. 

Different Users. Different Questions 

Stinging different users Into the question formulation stage can create 
additional challenges for the evaluator because different users may be 
Interested In entirely different questions. Conflicts may surface 
between different decision-making levels or branches of a JTPA program 
as to what Is truly Important to know about JTPA. For example, at the 
service delivery level, program staff may be more Interested In the 
Impacts JTPA Interventions are having on clients and business 
participants. (Are clients being placed effectively? Are businesses 
benefiting sufficiently to stay In the program?) Administrative users 
may be more Intrigued with studying the cost-effectiveness of JTPA, 
whereas political leaders may be more concerned with justifying public 
expenditures or meeting constituents' perceived needs. 

When state and SDA users are jointly involved In evaluation, there are 
potentially thornier Issues to resolve as to the focus of the 
evaluation. Since the state can ultimately sanction a poorly 
performing SDA, that SDA must be more directly and unyieldingly 
concerned with performance Issues. State JTPA policy-makers, on the 
other hand, may feel less compelled to examine Immediate performance 
outcomes and focus Instead on more long-term effectiveness measures of 
the program. The question formulation stage Ideally can provide an 
additional opportunity for Information exchange and accommodation 
between these two groups. 
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Narrowing the Focus of the Evaluation 

Once users and evaluation staff have generated sufficient evaluation 
questions, these questions need to be prioritized and the scope of the 
evaluation determined , according to the time and resources allotted. 
Even though they seem Important, some questions may need to be 
eliminated because discovering their answers will prove too time- 
consuming or costly. 

Attempting to answer too many questions In one evaluation effort Is a 
common pitfall. When the scope of evaluation work Is too grandiose, 
staff and other resources may be stretched too thin to produce a 
quality product. An overly ambitious scope of work not only Increases 
the complexity of coordination of staff activity, but also Increases 
the likelihood of missed time deadlines and budget overruns. For all 
these reasons, defining priority questions as early as possible creates 
an Important foundation for later evaluation activity. [NOTE: If 
users formulated questions some time previous to the current evaluation 
effort, evaluators need to check them for continued relevance and to 
consider new questions which may have arisen In the Interim.] 

Rating Evaluation Questions 

A JTPA evaluation might address any number of potential questions about 
client outcomes, employer participation, and JTPA program 
Implementation. Out of all the possible questions, the evaluation 
users must pare down thel r list to those questions which seem most 
essential and most feasible to answer. To prioritize and arrive at key 
questions, users may need to apply a rating system to each question, as 
the following sample shows. 



77 



EVALUATION QUESTION RATING SYSTEM 



Importance Feasibility 

What are participant wage levels, 

employment levels and welfare 
status 6 months after termination 
from JTPA? 

. How do employers benefit from 

participation In JTPA? 

How are service delivery arrange- 

ments supporting or Inhibiting the 
achievement of JTPA program goals? 



Instructions: Rate the Importance and feasibility of 

answering the above questions, using a scale of 1 to 5 
where 5 = high Importance or feasibility, and 1 = low 
Importance or feasibility. Those questions which are 
rated high In Importance (4 to 5) and are considered at 
least moderately feasible (3 or higher) should be 
considered for evaluation. 



Adapted from: Kay Adams and Jerry Walker, Improving the 
Accountability of Career Education Programs: Evaluation 
Guidelines and Checklists . The National Center for Research 
In Vocational Education, Columbus, OH, 1979, p. 38. 



Narrowing the evaluation focus to a specific set of questions to be 
answered can be one of the more frustrating and time-consuming steps In 
planning a JTPA program evaluation. The process may call for a 
generous dose of mediation and negotiation among different users. It 
may require the preliminary sketching out of various contingencies 
concerning funding, staffing and data collection and the revising of 
questions to meet these contingencies. This planning time Is well 
spent If It yields a manageable set of evaluation questions which 
reflect what users most want to know about JTPA. This set of questions 
forms the heart of the evaluation, Informing and directing the research 
efforts that follow. 
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WHAT h VTXON APPROACH MAKES SENSE? 

Once key valuation questions have been selected, the task 1s to choose 
a researcn strategy for answering those questions. The Issue at this 
stage of planning Is what strategy 1s most appropriate, given the 
nature of the evaluation questions and given numercus resource 
constraints, such as time, staff expertise, and data accessibility. 

Evaluation Approaches 

There are several basic evaluation research approaches and numerous 
variations on these approaches. Each approach has Its own strengths 
and weaknesses and Is appropriate to answering particular kinds of 
questions. This entire series on JTPA evaluation focuses on three main 
types of evaluation: n et Impact . gross Impact and process 
evaluations . For an overview of the Important characteristics of and 
differences between these evaluation approaches, refer to Chapter 2 of 
this volume and the Introductory chapters of Volumes III, IV and V. As 
a quick review, these three evaluation approaches are summarized below 
In terms of the sorts of evaluation questions to which each approach 
best responds. 



EVALUATION APPROACH: NET IMPACT 
General Questions Asked : JTPA-Spedf 1c Questions : 



What outcomes can be attributed 
to the program, rather than to 
other Influences? 

What service strategies are most 
effective for which groups of 
clients? 



Do JTPA clients In general do signifi- 
cantly better In the labor market than 
non-participants with similar profiles? 

What kinds of treatments have a greater 
Impact on client earnings? 



Are multi-strategy program approaches 
more 1 Ikely to have a greater Impact 
than single strategy programs? 

Do some client groups benefit from 
JTPA ( 1n terms of Increased < *irnings) 
than other client groups? 
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EVALUATION APPROACH: GROSS IMPACT EVALUATION 



General Qu estions Asked : 

What are the post-program 
outcomes for program 
participants? 



How are employers affected 
by the program? 



JTPA-Soec1f1c Questions ! 

What 1s the overall picture of 
participant employment, wages, and 
welfare status at three months, six 
months, or nine months after 
termination? 

How does this picture change over 
time? 



How do treatment results 
compare to one another? 



How may placements for clients 
be characterized? 



To what degree 
participation raise 
turnover rate for 
affect training 



does JTPA 
or lower the 
an employer? 
time? affect 



supervision or hiring? 

Which treatment strategies (e.g., 
long-term vs. short-term, OJT vs. 
class-room training) have more 
positive outcomes relative to other 
treatment strategies? 

Which treatment strategies are more 
effective (relative to other 
treatment strategies) for a given 
group? 

Do post-program jobs for JTPA clients 
resemble primary or secondary market 
positions? Are positions training 
related? 



EVALUATION APPROACH: PROCESS EVALUATION 



General Questions Asked : 

How 1s the program being 
Implemented? 

Is program Implementation 
affecting program outcomes? 



JTPA-Spedf1c Questlnns : 

How are JTPA policies being 
formulated and carried out? 

How are state policies and procedures 
affecting JTPA service delivery? 

Are certain state policies 
supporting or Inhibiting achievement 
of JTPA goals? 



SO 
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If the set of key questions selected straddles more than one evaluation 
approach, but comprehensive evaluation 1s not feasible, something must 
give. An obvious option 1s to pursue only those questions clustering 
around a single evaluation approach. However, this approach has its 
drawbacks: eliminating all process-related questions In favor of 
Impact related ones, for example, may ultimately narrow the utility of 
the Impact findings; the richness and explanatory capabilities of 
process findings are sacrificed. 

Alternatively, evaluation planners might contemplate multiple, but 
scaled down evaluation approaches to accommodate the various questions 
that are of greatest Interest. Both the gross Impact and the process 
evaluation designs lend themselves to this kind of flexible application. 

Obviously, a number of factors In the real world will influence the 
kind of evaluation approach selected: evaluation costs, timeframe for 
accomplishing the evaluation, data requirements, staff and other 
resource capabilities, and organizational demands. But regardless of 
these various considerations, the approach should be driven by 
questions of central importance to users and funders If the ^valuation 
findings are to benefit and be of use. 

WHAT DATA, DATA COLLECTION AND DATA ANALYSIS METHODS WILL 
BE REQUIRED? 

Settling upon a basic evaluation approach Is only the first step In a 
series of res^rch planning decisions about how the evaluation is 
actually to be v arr1ed out- the specific JTPA variables to be studied, 
the kinds of daia to be collected, and the manner 1n which the data are 
to be collected and analyzed. The end result of these decisions is a 
feasible resea rc h design for answering the questions initially posed 
about the efficiency or effectiveness of JTPA. 

Here anain, real world considerations impinge upon the choices 
evaluation planners would ideally like to make. The full range of data 
desired may be too costly or time-consuming to collect In its 
entirety. Some InTormatlon may be difficult to retrieve or 
Inaccessible. °taff may lack expertise in specific kinds of 
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statistical analysis required by a research approach. In recognition 
of these kinds of Issues, the specific analysis guides (Volumes III, 
IV, and V) for process, gross Impact and net Impact evaluations 
Attempt to balance the need for practical, flexible assessment tools 
with requirements for scientific soundness In the research methods 
used. Specific questions about the kinds of data to be collected and 
analyzed are addressed In each of these analysis guides. 

Some general data collection and MIS-related Issues cross-tut the 
various evaluation approaches. Is the requisite data available through 
the current Information system? Is the data comparable across SDAs? 
What kind of data sharing agreements across agencies will be 
necessary? These sorts of Issues will be covered in more detail In 
Chapter 8. They are only mentioned 1n passing here to emphasize that 
planning for data collection and analysis may Involve some special 
challenges to be discovered and met well In advance of evaluation 
Implementation. 

CONCLUSION 

The evaluation design process begins with a set of well-defined 
questions reflecting what administrators, funders or other users most 
want to know about JTPA programs. These questions, 1n turn, largely 
determine what the overall evaluation approaches will be. The task of 
the evaluation planner at this stage 1s to translate the general 
framework of evaluation questions Into a specific research design for 
accomplishing the evaluation. This task Is the central focus of the 
net Impact, gross Impact, and process evaluation guides 1n this 
series . 

The following chapters assume that planners have already considered the 
Important research design issues and are nov able tc move on to mon; 
specific resource and Implementation planning. 
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CHAPTER 6 
DEVELOP? ! EVALUATION PLAN 

What Dews v ••r-.jd Evaluation Plan Entail? 
What Resources Will Evaluation Require? 
What Time Schedule Will Evaluation Activities Depend Upon? 
How Will Evaluation Activities Be Monitored? 



CHAPTER 6. 



DEVELOPING AN EVALUATION PLAN 

INTRODUCTION 

Once the majtr evaluation research questions outlined In the previous 
chapter have been resolved (a research approach and design selected, 
data collection and analysis Issues resolved), evaluation planners can 
think more specif Ice . ;y about how the evaluation wl 11 be Implemented 
and can chart a course f ; the evaluation activities to follow. Thl s 
course of planning Is highlighted by the following questions: 

• What does a good evaluation plan entail? 

• What resources will the evaluation require? 

• What time schedule will evaluation activities depend upon? 

• How will evaluation activities be monitored? 

• What will the evaluation cost, and will potential benefits 
outweigh the cost? 

This chapter will tackle v/,e first four questions. The Issue of 
evaluation costs and benefits w1 11 be reserved for the following and 
final chapter of this section. 

WHAT DOES A GOOD EVALUATION PLAN ENTAIL? 

A written evaluation plan 1s an Invaluable tool for both 
conceptualizing and carrying out well-coordinated, timely, and useful 
evaluation activities . Ideally, an evaluation plan comprehensively 
documents all the various planning and management decisions which must 
precede and direct the actual carrying out of the evaluation. 

Committing this plan to writing 1s helpful 1n several ways. First, a 



written plan creates a conceptual record which can continually be 
referred to for clarification and direction. As a written record, the 
plan Is more subject to outside review, critique and revision than Is a 
set of plans carried around In someone's head. A written record also 
allows for a more broadly shared understanding of the evaluation 
process and how the conceptual work of planners will shape that 
process. For the evaluation team, of course, such an understanding Is 
crucial to the efficient coordination of evaluation tasks. Evaluation 
users may also appreciate knowing more about the planning 
considerations Influencing the evaluation, as documented In a good 
evaluation plan. 

The Plan as a Blueprint for Action 

Rather than a single document, the comprehensive evaluation plan can 
consist of a number of Interrelated statements, descriptions, charts 
and checklists. Informal notes, memos and Interviews can be supporting 
or supplemental documents to the main plan. 

Whatever written format Is used, the core of the plan should provide a 
detailed blueprint of the sequential activities occurring In each phase 
of the evaluation. The evaluation process usually encompasses three 
major phases: 

• a planning phase 

• an Implementation phase 

• a reporting and dissemination phase 

Since the activities each phase Includes will vary from one evaluation 
setting to another, no set checklist of activities can apply to all 
situations. The evaluation activities listed on the following page are 
meant to Illustrate the generic categories of activities a plan might 
cover. 



A SAMPLE LIST OF EVALUATION ACTIVITIES 
COVERED IN AN EVALUATION PLAN 



PHASE I: PLANNING 



(1) Collecting background Information on JTPA programs, Including: 

• Reading and analyzing relevant program related documents , past 
reports 

• Preliminary meetings with sponsors and other users of evaluation 

• Preliminary Introduction/briefing with program staff 

• Site vlslt(s) 

• Selecting advisory committee 

(?) Assessing evaluablllty 

• Interviewing key staff regarding technical, organizational , and 
political factors affecting evaluablllty 

• Brief outline of findings and recommendations for proceeding 

• Meeting with and feedback from program administrators 

(3) Formulating questions 

• Review by users and advisory committee 

• Question and answer session with users (feasibility Issues) 

• Final selection of questions 

(4) Developing an evaluation research design 

• Review data to be collected (availability, validity, reliability) 

• Data collection procedures (sampling strategy and Interview 
procedures) 

• Data analysis procedures 

(5) Assigning and briefing staff and developing an overall resource plan 

(6) Developing dissemination strategies 

(7) Reviewing by advisory committee 
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EVALUATION ACTIVITIES, continued 



PHASE II: IMPLEHENTATION 

(1) Briefing all concerned staff 

(2) Field testing Interview Instruments 

(3) Data collection 

• HIS data 

• Interview data 

(4) Data cleaning procedures and other preparation for analysis 

(5) Analyzing data 

(6) Interpreting the results 

PHASE III: DISS EH I NAT I ON 

(1) Preparing Interim reports 

(2) Reviewing by users and advisory committee/questions and 
feedback 

(3) Preparing final report and recommendations 

(4) Reviewing (formal) 

(5) Preparing article-length summary of evaluation report 

(6) Scheduling question and answer meetlng(s) and final In-person 
presentation 



In serving as the evaluation's blueprint, the core of the evaluation 
plan covers not only activities , but also the costs , timing, resources 
and management which these activities entail. The part of the 
blueprint which focuses on resource utilization and costs Is sometimes 
called a resource plan, which 1s the focus of the next section. (A 
specific example of a resource plan 1s presented In the next section.) 

Statement of Purpose 

: n addition to a blueprint for action, the overall plan should contain 
a . v fMient of the evaluation 1 s purposes and goals and the questions 
ths luation Intends to address. Such a statement acts as a 
cor :.. Vi reference point for the rest of th* evaluation plan. At the 
erK> <:-' \ .: &*o luation , the statement of purpose also offers a yardstick 
for t:2c. •'rv'js the evaluation's accomplishments. Did the evaluation 
efft i stick to the original goals? Did It serve the purposes It was 
supposed to serve? Did it answer the questions that were posed? 

Summary 

There is no simple recipe for creating a good evaluation plan. From 
the preceding discussion, several guidelines may be distilled: 

• The plan should be In writing. 

• The plan should be comprehensive. 

• The plan should include a blueprint for carrying out all phases 
of the evaluation. 

• The plan should cover all evaluation activities, costs, timing, 
resources and management. 

• The plan should contain a ; ratement of purpose and goals. 

A s pecific check 11 rt for elements in the evaluation plan Is included at 
the end c? th*5 chapter. First, we look more specifically at some 
resource management aspects of the overall plan. 

*IHAT RESOURCES WILL THE EVALUATION REQUIRE? 

Since evaluation neeHs, interests and capabilities will vary from state 
to state, sc too will resources required. A r esource plan , a written 
strategy for accomplishing the evaluation, Is an essential tool for 



effectively planning and managing the evaluation effort. The plan may 

begin as a tentative document subject to all kinds of attacks and 

revisions In the Initial stages of evaluation planning. Before the 

actual evaluation focus (which questions are to be answered) and 

approach (what evaluation design Is appropriate) are delineated, the 

plan must be sketchy. But as certain early decision points are 
reached, the plan takes on greater detail and forn;. 

Elements of a Resource Plan 

A resource plan can be devised according to a number of formats. 
Whatever format is chosen, the basic elements of the plan Include: 

• A sequential listing of evaluation tasks to be performed and 
products tc be produced 

• A time allotment for each task 

• The staff and other resources needed for each task 

« An estimate of the quantity or 3&ount of resources requi red 
(number of staff hours, computation or word processing time, 
etc.) 

All of the above elements need to be Identified iji writing and combined 
In some easily readable form. A simplified example of an evaluation 
resource plan follows , As this example Illustrates , many JTPA 
evaluatois j\ 1 1 require some special staff or consultant Input at key 
junctures. Tor a look at the sped*! staff skills JTPA evaluation may 
call for and other staffing Issues, see Chapter 10. 



SAMPLE RESOURCE PLAN FOR EVALUATION 



Activities 



PHASE I: Planning 

(1) Collecting background 
Information 



(2) Assessing evalu- 
ablllty 

(3) Formulating 
questions 

(4) Developing a 
design 



(5) Assigning and 
briefing staff/ 
developing resource 
plan 

(6) Developing dissem- 
ination strategies 

(7) Reviewing and feed- 
back by Advisory 
Committee; making 
revisions 



Staff staff Total other Special 
Assignments Time Time Staff and 

(days) St f /Other Resources 



Sanchez 
Johnson 
Heller 

Johnson 



Johnson 
Heller 

Johnson 

Chang 

Killer 



Sanchez 
Johnson 



Sanchez 



Sanchez 
Johnson 



2 

10 

5 



5 
2 

2 
7 
7 



2 
2 



17/3 

2/2 

7/1/2 

16/3 



4/0 



1/0 
6/1 



consultant (3 days) 



consultant (2 days) 



consultant review 
(1/2 day) 

borrow statistician 
from agency X for 
review (1 day) ; 
consult with pro- 
grammar (2 days) 



advisory 

committee*; consul- 
tant review of plan 
(1 day) 



PHASE II: Implementation 

(1) Briefing affected Heller 
program staff Johnson 

(2) Conducting field test Miller 
of Interview Instrument 

(3) Data collection Chang 

Miller 



2 4/0 
2 

3 3/4 field test Inter- 

viewers (2 days) 

10 16/14 Interviewers (10 
6 days); computer 

time and operators 

(4 days) 



* Advisory committee time Is not calculated here because 1t 1s an external 
resource which 1s free to the evaluating agency. 



(SAMPLE PLAN, continued) 



'4) Reviewing data 

cleaning procedures/ 
Preparing for analysis 

(5) Analyzing data 



(6) Interpreting results 



Chang 



Chang 

Miller 

Johnson 



Chang 

Miller 

Johnson 



2 
2 
2 



4/1 consult with pro- 
grammar (1 day) 



15/f computer time (2 
days; borrow 
statistician for 
review (2 days) 

6/1 consultant review 
(1 day) 



PHASE III: Dissemination 

(1) Preparing Interim Chang 3 5/2.5 
Reports Johnson 2 

(2) Reviewing by users/ Johnson 1 1/0 
committee 

(3) Preparing final Chang 4 8/5 
report Johnson 4 



(4) Reviewing Johnson 1 1/0 

(5) Preparing special Johnson 2 2/0 
summary 



word processing 
time (2.5 days) 

advisory coniiiitl 
(1 day) 

word processing (3 
days) and public 
affairs (2 days) 

advisory committee 



(6) Presenting In person 



Chang 



1/0 



Adapted from: Arlene Fink and Jaquellne Kosecoff, Evaluation 
Basics : A Practl t loner 1 s Manual . Beverly H11 1 s „ CA: Sage 
Publications, 1982, p. 35. 



The utility of a Resource plan 

The resource plan 1s important and useful to the evaluation effort for 
a number of reasons: 



A thorough resource plan anticipates all activities and tasks 
comprising toe evaluation and the kinds of resources necessary 
for the completion of those tasks. 

In apportioning out the work to be done, a resource plan can 
suggest a realistic timeframe for accomplishing the evaluation. 

The plan may encourage comparison of alternative allocations of 
resources. 



The plan Identifies resource gaps which may need to be filled 
by outside consultants or others. 

The plan permits administrators to appropriately plan for and 
coordinate the use of special resources, such as extra 
technical expertise which may be difficult to obtain on short 
notice. 



The plan acts as an ongoing management tool for tracking and 
coordinating multiple activities. 



WHAT TIME SCHEDULE WILL EVALUATION 
ACTIVITIES DEPEND UPON? 

As with any project work plan, the evaluation resource plan should also 
Include a specific schedule for the accomplishment of tasks. The 
scheduling tilinenslon Is Important to the evaluation effort for a number 
of reasons: 



Evaluation Timing an d User Commitment : if not accomplished within a 
specified timeframe, evaluation results can go stale. The 
organizational momentum behind evaluation may die and the results, when 
f'nally produced, may no longer be valued or utilized. Over a period 
of time, the potential users of the evaluation may change 
substantially. New users may have less commitment or interest 1n the 
evaluation or may feel more threatened by the Information evaluation 
elicits. For these reasons, user input may Inform the scheduling , as 
well as content of the evaluation. 



The Time Schedule as a Manage ment Tool : Establishing a timeframe Is 
also critical to the day-to-day management of the evaluation. Careful 



areas and resource bottlenecks, will lead to more efficient resource 
utilization. A detailed timeframe also acts as a monitoring tool for 
keeping task accomplishment on schedule. However, the timeframe Is 
only as good as the component task Information of the resource plan. 
The more sketchy the resource plan, the more difficult to realist^ally 
allocate time and sequence evaluation activities. The evaluation 
resource plan (Introduced earlier) can be easily expanded to Include 
more specific scheduling Information for managing the evaluation: 



SAMPLE 


RESOURCE PLAN 


WITH TIMEFRAME 








Staff 


Staff 


Total 


Activities 


Dates 


Assignments 


Time 


Time 








(days) 


Stf/Other 


PHASE I: Planning 










(1) Collecting background 


2/1 - 2/10 


Sanchez 


2 


17/3 


Information 




Johnson 


10 








Heller 


5 




(2) Assessing evalu- 


2/11 - 2/13 


Johnson 


2 


2/2 


ablllty 








(3) Formulating 


2/13 - 2/18 


Johnson 


5 


7/. 5 


questions 




Heller 


2 





Time Schedule as a Coordination Tool : The scheduling of an 
evaluation should also mesh with relevant funding, legislative and 
planning timetables. For example, evaluation findings with 
Implications for broad policy-making might Ideally be coordinated with 
the policy timeframe of the State Coordinating Council. Evaluation 
plans might also be coordinated to Inform allocation decisions for 
state set-aside monies or other administrative actions. The Important 
point In overall scheduling Is to seize, wherever possible, Important 
coordination opportunities with other actors within the total JTPA 
system. Such coordination can only enhance the ultimate utility of the 
evaluation effort. 



HOW WILL EVALUATION ACTIVITIES BE MONITORED? 
In scheduling evaluation activities, planners can build Into the 
evaluation process opportunities for review, comment, and revision. 
These opportunities for monitoring significant phases of the evaluation 
can enhance the overall evaluation effort In several ways: 

Review opportunities build flexibility Into the evaluation 
plan, allowing for changes and Improvements where necessary. 

• Review allows for alternative decision pnints to be scheduled 
Into the evaluation process rather than forcing a decision 
before adequate Information Is available. 

• Review, In encouraging the timely discovery and correction of 
research problems or planning gaps may ultimately save time and 
resources . 

• External review by an Independent third party can Increase the 
user's confidence In and n yerall credibility of an in-house 
evaluation. 

Review can be scheduled not only for the early planning phases of the 
evaluation, but also following later phases of Implementation and final 
reporting. This kind of more comprehensive review offers Insights Into 
the strengths and weaknesses of the evaluation upon which 
recommendations for future evaluation activities can be based, (for 
more on formal evaluation review, or audit , see Chapter 10.) This 
chapter concludes with a sample checklist for reviewing an evaluation 
plan. 

Reviewing Evaluation Plans 

In concluding this chapter with a plan review checklist, we come back 
full circle to the Inltla' question posed: What does a good evaluation 
plan enlall? The review example below suggests four separate frame- 
works for assessing the adequacy of a plan: conceptual . 
organizational . r esearch , and management . 
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COMPONENTS IN AN EVALUATION PLAN 



Instructions : Rate your evaluation plan by checking the appropriate 
descriptive category for each component of the 
written plan. 



Conceptual Framew ork 

1. Statement of purpose(s) 
for the evaluation? 

2. Questions to be .addressed? 

3. Users to be served? 

4. Potential users of the results? 

5. Overview of evaluation approach, 
research activities? 

6. Evaluation products expected? 



Well Partially Not Not 
Defined Defined Defined Appl 1 cable 



Organizational Framework 

7. Methods for assessing evaluablllty? 

8. Strategies for Increasing leadership and 
organizational support for evaluation? 

9. Organizational factors affecting the 
location of evaluation? 

a. Authority and compliance factors? 

b. Credibility factors? 

c. Neutrality and independence factors? 

d. I>chn1c<vl and other competency 
factors? 

e. Coordination capability factors? 



PLAN COMPONENTS, Continued) 



Well Partially Not Not 
Defined Defined Defined Appllcat 

10. User Involvement and feedback? | | |' | | | i — i 

11. Educational and briefing activities? |~! |~l l~l iZj 

12. Advisory group participation? |~| |~| | | | | 

13. Community participation, community 

resource utilisation? | | |~j | — | i — i 

14. Evaluator role and responsibilities? |~| |~| |~| i i 

15. Program staff roles and responsibilities?' |~| |~| |~| |~| 

16. Mechanisms for interim feedback to _ 

users and program staff? j | j |_| | | 

17. Strategies for enhancing |~| |~| | | | | 
staff cooperation? — 

18. Intra- and Interagency agreements for _ 

data/resource sharing? | _j | | | *| i i 

19. Dissemination strategy? |~| |~| | j j | 

20. Confidentiality agreements and staff _ 

protections? | | | | i i i — i 

Research Framework 

21. Theoretical basis for research design? |~| |~| |~| |~| 

22. Data gathering Instruments? t J l~! I~l |~| 

23. Data gathering procedures? ;" |~| ( | I I 

24. Sampling strategy? |~| |~| j~| | | 

25. Data storage <md retrieval procedures _ 

(Including data merging procedures)? | | | | | | |~| 

26. Procedures for reviewing data _ 

reliability , validity , comparability ? |_| |~| |~| |~| 

27. Data analysis procedures? |~l iZj l~l I I 

28. Data Interpretation methods? |~| |~| j j | | 

ERIC 



PLAN COMPONENTS, Continued) 



Well Partially Not Not 
Management Framework Defined Defined Defined Applicable 



29. A dissemination plan for findings? 

30. A plan for Interim reports, briefings? 

31. Sequential list of all evaluation 
planning tasks and activities? 

32. Sequential list of all Implementation 
tasks and activities? 

33. Sequential list of dl'l reporting 
and dissemination activities? 

34. List of all products to be produced? 

35. A timeframe for tasks and products 
completion? 

36. Staff and other resources (facilities/ 
equipment) needed for each task? 

37. Procedures for contracting with a 
consultant? 

38. Who will perform various tasks? 

39. Job qualifications and job descriptions 
for staff? 

40. Review procedures? 

41. Policies and procedures affecting the 
evaluation? 

42. Evaluation costs and benefits? 

43. Budget allocations? 



Adapted from: Kay Adams and Jerry Walker, Improving^ the Accountability 
of Career Education Programs: Evaluation Guidelines and Checklists . 
Columbus, OH: The National Center for Research In Vocational 
Education, 1979, p. 69. 
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CHAPTER 7 

ESTIMATING EVALUATION COSTS AND BENEFIT? 

What Will an Evaluation Cost? 
Can Evaluation Benefits Be Measure'"? 
Will Benefits Outweigh Costs? 



98 



CHAPTER 7. 



ESTIMATING EVALUATION COSTS AND BENEFITS 



INTRODUCTION 

These volumes on evaluating JTPA are premised on the notion that 
evaluation Is an important management tool for decision-makers f and 
offers key benefits in terms of Improved understanding and operation of 
JTPA. While perhaps easily accepting this premise In the abstract, 
state JTPA administrators and other decision-makers will want to know 
the bottom line In more concrete terms before committing to 
evaluation. How much will evaluation cost, and will the purported 
benefits outweigh the costs? The answers to such questions are usually 
not neat and straightforward: the benefit-cost calculation Is often 
very elusive. In this chapter we examine briefly some of the Issues 
associated with estimating the costs and benefits of evaluating JTPA. 

WHAT WILL AN EVALUATION COST? 

As Is the case with any plan, estimating the costs of evaluation Is a 
critical step In the planning process. Funders need a preliminary 
price tag before authorizing an evaluation effort, and as early as 
possible evaluation planners themselves will want to anchor evaluation 
options to concrete financial realities. The thorough costing of the 
major evaluation components provides a realistic basis for comparing 
evaluation alternatives and assessing the relative merits of different 
data collection and staffing strategies. An estimation of costs and 
benefits encourages planners to creatively rethink alternative resource 
and staffing strategies or consider one or more scaled-down versions of 
the preliminary evaluation design. 



Evaluation costs will vary tremendously depending on the purpose and 
scale of the evaluation effort, the kinds of resources an organization 
can marshal to do the evaluation, and the existing market cost for 
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external resources, such as consultants. For example, consultant fees for 
an evaluation specialist may range from $100/day to *600/day, or more. 
Personal field Interviews can cost from less than $100 to more than $50(5 
per Interview, depending on consultant fees and how difficult It is t 
locate an Interviewee and collect the Information. Sometimes reduced U 
or In-kind contributions are available, altering the cost framework for 
evaluation substantially (see Chapter 9 on alternative funding options). 

The preliminary resource plan provides a ready format for assessing 
evaluation costs. To the evaluation activities, schedules and resources 
columns Is added an additional column for costs, as excerpted below: 





SAMPLE RESOURCE 


PLAN WITH COSTS 








Staff 


Total 


Other Special 






Activities 


Time 


Time 


Staff and 


Costs 






(days) 


Stf/Other Resources 




PHASE I: Planning 










(1) 


Collecting back- 


2 


17/3 


consultant 


Consultant: 




ground Information 


10 




(3 days) 


$250/day @ 






5 






3 days = $750 


(2) 


Assessing evalu- 


2 


2/2 


consultant 


Consultant: 




ate 11 ty 






(2 days) 


$300/day @ 












2 days = $600 


(3) 


Formulating 


5 


7/1/2 


consultant 


Consultant: 




questions 


2 




review 


$250/day @ 










( .5 days) 


.5 days=$125 


(4) 


Developing a 


2 


16/3 


borrow statistician 


Agency statis- 




design 


7 




from agency X for 


tician: $30/ 






7 




review ( 1 day) ; 


hr x 8 = $240 










consult with pro- 


(agency rate) 










grammar (2 lays) 




(5) 


Assigning and 


2 


4/0 




Agency pro- 




briefing staff/ 


2 






grammer: $30/ 




developing 








hr x 16 = $480 




resource plan 








(agency rate) 



[Note: These hypothetical costs cited are only given as a general example 
of how costs must be linked to specific evaluation activities listed In a 
resource plan. The figures do not reflect actual costs and should not be 
taken as representative of evaluation costs In general.] 
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Counting All Evaluation Costs 

The above example of evaluation costs is overly simplified in that U 

only lists obvious extra costs such as consultants. A truly effective 

cost assessment must Include alj costs borne bjr the sponsoring agencies 
1 3 

or agency, not just explicit dollar costs. Where In-kind resources 
such as staff time, computer time, administrative overhead and 
materials are shifted to an evaluation project, those resources should 
also be fully costed out. In such cases, It may be more convenient and 
meaningful to cost out some resources In other than dollar terms, such 
as staff hours to be donated to the evaluation. (Examples of various 
evaluation costs appear on the following page.) 

Less Quantifiable Costs 

Quantifiable costs, such as labor and materials, are only part of the 
total cost eguatlon. These costs must be considered In concert with 
other, less definable costs. Examples of this more elusive category of 
costs might Include the level of anticipated program disruption caused 
by the evaluation or resource losses associated with an inexperienced 
staff. 

Perhaps some of these non-guantif lable costs can only be compared 
across different evaluation strategies In term:* of the negative impacts 
on utilization . Consider the strategy of using In-house staff vs. 
outside consultants. In some cases, the former strategy may be much 
cheaper, but the results less credible to important funders or 
decision-makers. While not measurable, the potential costs of reduced 
credibility and utilization are nonetheless Important to the overall 
cost calculus. The chart on the following page categorizes the various 
potential costs, both guantlflable and not-so-guantlf lable, associated 
with evaluation. 



Evaluation theorists hold divergent notions as to how costs should 
be calculated. See for example, Scrlven, Michael , "Costs In 
Evaluation" 1n The Costs of Evaluation by Marvin C Alkln and 
Lewis C. Solmon, eds. (Beverly Hills, CA: Sage Publications, 
1983) pp. 27-44. 
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THE COSTS OF EVALUATION 



Quantifiable Costs 
Direct Costs 



Indirect Costs 



Travel 

Evaluation staff salaries/benefits 

Consultant fees 

Per diem expenses 

Telephone and mall 

Computer time for data 

processing 

Printing/duplication 
Published materials 
Supplies 



Overhead 

• Facilities and space 

• Equipment rental, use and repair 

• Utilities 

• Administrative time 

Support Services 

• Secretarial/office 

• Accounting 

• Legal (e.g., contracting, client 

confidentiality Issues, data 
use Issues, etc.) 

• Public relations 

• Publishing 



Non-Quantifiable Costs 

Potential Costs to 
Staff and Clients 



Interagency coordination costs 
Program disruptions 
Service Inefficiencies 
Interview time 



General Program- 
related Costs 
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Credibility problems and costs 
Mistakes, Inefficiencies of 

Inexperienced staff 
Time delays 

Staff resistance to evaluation 
Inadequate or Inappropriate 

utilization of evaluation 

results 
Political costs 



CAN EVALUATION BENEFITS BE MEASURED? 

The costs of various evaluation strategies are most meaningfully 
Interpreted In the context of comparative benefits to be derived from 
each strategy. However, evaluation benefits are far more resistant to 
comparative calculation than are costs. First, most potential benefits 
of evaluation are more difficult to i.uasure or are Intangible. The 
primary benefit of evaluation 1s better Information about JTPA, but 
whether that Information Is well-utilized and leads to program 
Improvements Is another question. After-the-fact program Improvements 
may be translated Into quantifiable program gains (more clients 
referred, more clients served), but no such calculation can be made 
Prior to the evaluation. Second, the potential benefits of evaluation 
are often long-range and difficult to predict, not only In terms of 
degree of benefit, but also In terms of who will benefit. The benefits 
to be derived from evaluating a currently successful program may 
largely accrue In the future to entirely different programs 1n 
different local or state settings. 

Finally, evaluation may confer on an organization secondary benefits 
which are often not considered In the benefit-cost equation because 
they are by-products of the evaluation process rather than directly 
related to the evaluation findings. The following section discusses 
the notion of Indirect benefits further. 

Direct and Indirect Benefits of Evaluation 

Anticipated central benefits of JTPA program evaluation will most often 
relate to better Information leading to future Improvements In program 
efficiency and effectiveness . These direct benefits of evaluation are 
explored In some detail In Chapter 1, In addition, the evaluation 
process may lead to certain organizational enhancements, or Indirect 
benefits . which are not explicitly connected to JTPA goal 
achievement. For example, evaluation planning may result In better 
Inter- and Intra-agency communication and/or coordination In areas 
beyond JTPA evaluation. Evaluation Implementation may result In an 
enhanced HIS or other data collection Improvements. Examples of the 
various potential benefits (both direct and Indirect) to be derived 
from evaluation are summarized on the following page. 
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BENEFITS OF EVALUATION 



Direct Benefits to « Improved understanding of JTPA 

JTPA Programs program activities and outcomes. 

• Increased accountability to program 
funders/publlc. 

• Recommendations for Improved program 
efficiency and effectiveness. 

Information for JTFA planners 
and managers 
° Information for JTPA policy- 
makers 

Information which complements 
and moves beyond performance 
standards 



Indirect Benefits to 
JTPA Organizations 



Indirect Benefits to 
Ovher Programs and 
Individuals 



Indirect Social 
Benefits 



Improved Intra- and Interagency 
communication/coordination . 

New contacts within the research 
and professional communities. 
Enhanced "partnership 11 with 
business and professional groups. 

New funding connections and 
capabilities. 

Improved capabilities for doing 
future evaluation, Including 
Improved program evaluabll Ity. 

Enhanced MIS or other data 
collection systems. 

Improved data cleaning procedures 

Increased political credibility. 



Lessons learned from one evaluation 
setting applied to other settings. 

Improved services to the Intended 
target groups. 



Increased public awareness of and 
support for JTPA. 
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WILL BENEFITS OUTWEIGH COSTS? 

Those who are looking for concrete benefit- cost decision rules for 
doing (or not doing) a JTPA evaluation will remain disappointed. We 
can take some comfort In the fact that cost factors are relatively 
discrete and quantifiable, allowing decision-makers to more readily 
compare costs of competing evaluation alternatives (and, of course, 
competing non-evaluation uses of resources). The difficulties come In 
plugging evaluation benefits Into the equation; how can one assign 
measurable value to the various Informational and organizational 
benefits a JTPA evaluation can yield? Evaluation clearly does not lend 
Itself to any straightforward balancing of numerical costs and benefits 
to see which outweighs the other. 

However, the inability to assign costs and benefits along the same 
quantitative dimensions does not preclude the use of cost and benefit 
information in choosing whether and/or what kind of evaluation 
alternative to pursue. Even If evaluation benefits are more 
subjectively assessed, It is still Important to establish how 
evaluation costs stack up against those benefits. As with costs, the 
resource plan provides the starting point for developing a checklist of 
benefits. ( Direct benefits being related to the kinds of Information 
outcomes provided by the evaluation and Indirect benefits resulting 
from the evaluation process Itself, as discussed above.) 

If alternative evaluation strategies are being considered, a thorough 
checklist of benefits for each alternative provides a richer context 
for weighing costs. In order to more closely compare different 
clusters of benefits , decision-makers can assign weights to each 
benefit as a rough way of measuring each benefit's Intrinsic value to 
the evaluation user(s) . Each evaluation alternative could then be 
measured In terms of a total benefit "score", as well as total 
evaluation cost. The more costly evaluation alternative may provide 
unique and highly valued benefits which significantly outstrip the 
potential benefits offered by less costly approaches. 

CONCLUSION 

The cost of an evaluation is an Immediate an inescapable concern for 
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The JTPA planner and decision-maker. Is the evaluation doable, or will 
It cost too much? Often, however, this full scrutiny of evaluation 
costs Is not equally applied to evaluation benefits. Although benefits 
may be less quantifiable and more subjectively felt than costs, they 
are nonetheless real, substantial and Important In providing a fuller 
context for assessing costs. 

In assessing costs and benefits, planners have to remain open to 
creative alternatives for carrying out a JTPA evaluation so that they 
do not feel locked In to a single, too costly plan. The next section 
explores some JTPA evaluation staffing and funding alternatives. 
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SECTION 4 
IMPLEMENTATION ISSUES 



The preceding chapters trace the JTPA evaluation planning process, blending 
implementation issues into that process. How the evaluation will be conducted, 
who will be involved, what information will be gathered— these are all implemen- 
tation issues that must be imbedded within the overall evaluation plan. The 
separation between planning issues and implementation issues is a somewhat 
artificial one, made here for clarity's sake. This final section pulls out three critical 
areas of evaluation implementation for closer examination: data collection issues, 
staffing concerns and funding options. 
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chapter 8 
collecting and using data 



What Is fho Quality of the Data? 

Data Reliability 
Data Validity 
Data Comparability 

Are the Data Available? 

Will Different Data Sets Need to Be Merged? 

How Will Client Confidentiality Be Handled? 
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CHAPTER 8. 



COLLECTING AND USING DATA 

INTRODUCTION 

Whether data are derived from an MIS or other automated 
systems, access to accurate and valid data Is a key consld 
designing and Implementing any evaluation. Without adequate 
most beautifully designed evaluation Is worthless. Evaluati 
not wait until the final design and Implementation sta 
evaluation to plow through data gathering systems and be 
with their Inadequacies. Rather, these systems should be ex 
their Insufficiencies uncovered In the early evaluation plannl 

Many considerations besides analytical needs (e.g., 
technical, ethical) go Into the design of a data collection s 
a result, each system uniquely delimits what Informi 
immediately feed into evaluation. Given this diversity 1 
other data systems across states and SDAs, the purpose In th 
is to highlight those major data Issues relevant to i 
programs. As with other evaluation concerns presented In th 
data collection Issues have not only a technical face 
organizational face as well. Each of the following issue 
discussed In turn.: 

• What is the quality of the data? 

• Are the necessary data available? 



A more detailed discussion of HIS capabilities Important to evaluation 
activities will be presented 1n a later publication 1n this series of 
volumes . 

WHAT IS THE QUALITY OF THE DATA? 

For both monitoring and evaluation purposes, a primary concern is the 
quality of the data. Quality rests principally on the reliability , 
validity and comparability of the MIS and other data sets to be used In 
evaluation. 

Data Reliability 

Reliability has to do with the accuracy and consistency with which data 
have been collected. In the HIS, for example, there are several major 
sources of unreliable data: (1) the client himself or herself; (2) the 
staff who are recording Information on the client; (3) the data entry 
staff transferring that Information; (4) system classification schemes 
which do not clearly or consistently distinguish one data element 
category from another. In SDAs with highly decentralized Intake and 
service delivery systems, the potential for data Inconsistencies and 
Inaccuracies Is multiplied. In preparation for evaluation, planners 
can review data collection procedures and safeguards, recommending 
additional safeguards If necessary. 

Data Validity 

A related Issue Is that of measurement validity : Do the data elements 
required In the evaluation truly measure what they are supposed to 
measure? For example, do simple "wages" truly represent "earnings" 
(the outcome JTPA legislation mandates for study)? If data on wages 
alone Is used as an outcome measurement, other earnings , such as fringe 
benefits and tips, may be Ignored. The analysis guides for process, 
gross Impact, and net Impact evaluation (Volumes 3, 4, and 5) deal 
further with validity Issues In the specific measurement context of 
each approach. 

Data Comparability 

Data collected within a single state or SDA may sufficiently meet 
standards of reliability and validity but still not be useful for JTPA 
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program comparisons across states and SDAs. In order to evaluate 
Implementation practices and outcomes within a broader state or 
regional context, the definitions of various HIS data elements need to 
be reasonably standardized. Achieving such standardization across 
different jurisdictions often proves to be a complicated task. 

A few states (e.g., Texas) have highly centralized HIS systems which 
eliminate basic concerns about retrieving common evaluation data from 
each SOA. However, most states have more decentralized systems In 
which SOAs operate their HIS with varying degrees of Independence from 
each other and from the state. The more decentralized and Independent 
SOA Information systems are, the more likely Issues of data 
comparability will challenge statewide evaluation planning efforts. 

It Is In the comparison of different program service strategies or 
treatments that MIS data comparability Is often most questionable. 
National reporting requirements have led to states and SOAs collecting 
fairly standardized Information about JTPA enrollments, terminations 
and primary client characteristics. But because such reporting 
requirements are lacking for program variables (e.g., type of 
treatment, length of treatment), treatment data are much less uniform 
across both states and SDAs. 

When no set statewide HIS definitions and coding guidelines exist, the 
definitions for various program treatments may be applied 1n 
non-standardized ways. Consider the category "pre-employment 
training." One SOA may lump Into this category clients who are given a 
half-day course on job search techniques along with clients taking a 
comprehensive three-week course. Another SOA may categorize only the 
latter activity as "training", and regard the first activity primarily 
as "placement. " This comparability problem can extend down to the 
Individual service provider level In a highly decentralized SOA where 
services are largely contracted out to numerous providers. 

Multiple treatment strategies may further complicate matters because of 
the added problem of defining which category gets credit for a 
resultant positive outcome. Crediting only the final treatment, as do 
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some SDAs, leads to a distortion of program outcomes: the success rate 
of the final treatment (often OJT) may appear greater than 1t actually 
merits because the costs per placement may be artificially deflated. 
Conversely, the costs per placement for all the more preparatory kinds 
of treatments (such as adult basic education, or skills training) may 
be over-Inflated, making these treatments appear less attractive. 

If HIS coding for these kinds of treatment variables are not 
standardized statewide, a state wishing to Include such variables In 
evaluation will have to establish clear guidelines for assigning 
treatment data to categories. SDAs may then be able to translate their 
own coded data more appropriately to fit statewide definitions. 

ARE THE DATA AVAILABLE? 

In any state setting, the HIS, providing continuously generated 
Information on a number of Important client and Implementation 
variables, will be a key factor In the evaluation. Besides data 
quality and comparability, a primary concern must be MIS sufficiency to 
meet the Important data requirements of evaluation. What demands, In 
fact, will evaluation place on the HIS? The different evaluation 
approaches presented In this series have different Information 
requirements which are detailed In each of the separate volumes of JTPA 
Evaluation at the State Level . ( In addition, a more specific 
discussion of the kinds of HIS capabilities which are desirable for 
JTPA evaluation will be provided In a forthcoming paper of this 
project.) 

In general , however, the various evaluation approaches wl 11 require 
many of the following basic categories of HIS data: 
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MIS DATA FOR EVALUATION 



1. Client characteristics 

• Age 

• Sex 

• Race 

• Etc. 

2. Service data 

• Type of treatment 

• Length of treatment 

3. Termination data 

4. Follow-up data 

• Client data 

• Additional services 

5. Employer data 

• Employer I.D. Information 

• Employer services Information 

• Employer follow-up 

6. SDA/Communlty characteristics 

7. Financial data 



If the HIS lacks certain data elements useful to evaluation, how 
readily can the system be revised? It may be more cost-effective In 
the long-run to hammer out a thorough revision based on multiple 
evaluation uses, rather than slowly attack a system piecemeal . 
However, In states where SDAs operate Independent software or mainframe 
systems, statewide modifications to such systems will be much more 
costly and time-consuming. SDAs, If bearing a proportionate share of 
those costs, may resist such modifications unless they can see clear 
benefits for local level analysis from these modifications. 

Computer programming time Is not the only cost Issue Involved In 
acquiring new data for evaluation. States need to be sensitive to the 
potential burdens that added reporting requirements will place on SDAs 
and service providers (designing new forms, training Intake personnel, 
etc.). Also, there 1s a limit to how much research Information an SDA 
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or service provider can collect without compromising Its social service 
mission. SOAs may have different Information priorities from each 
other and from the state, complicating the task of developing a uniform 
set of evaluation variables. Some states have successfully resolved 
5tich an Iss'je by negotiating changes In proposed statewide evaluations 
to Include gathering more Information of direct concern to SOAs. 

WILL DIFFERENT DATA SETS NEED TO BE MERGED? 

While HIS Information will often be at the core of many JTPA 
evaluations, additional Information may also be critical. For example, 
merging HIS client data with other kinds of client data on post-JTPA 
earnings, employment and welfare dependency permits a more 
sophisticated analysis of program outcomes and Impacts. 

Frequently this additional kind of data Is contained In data base 
systems completely separate and Incompatible with the JTPA HIS. The 
evaluation plan should anticipate the technical difficulties to be 
overcome In bringing various data systems together for a unitary 
analysis. 

Technical difficulties In merging data are not confined to the managing 
of different computer systems and programs; the data Itself may present 
stumbling blocks. For example, In some states the category 
"disadvantaged" Is not flagged In the Employment Service registrants 
data base used In the net Impact evaluation to construct a comparison 
group. Lacking this category, It will be more difficult to match and 
compare JTPA participants with similar groups of non-participants. 
(See Volume 5, Chapter 4 for an In-depth treatment of this concern.) 

The task of merging HIS with other kinds of data can Involve 
organizational considerations, as well. The data may be under another 
agency's authority, and obtaining that data may pose additional 
challenges. Commonly, data requests across agency boundaries are 
viewed as an Imposition, requiring extra staff time or other 
resources. If the lines of communication between agencies are poor, 
the data collection effort may suffer. 
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Such realities underscore the need for strategic organizational 
planning as part of the overall evaluation planning effort. 
Representatives of affected agencies should be brought Into the 
planning process early to ensure greater cooperation. Any Interagency 
understandings about data sharing and computer use should be put 1n 
writing as further Insurance against future frustrations and 
misunderstanding. 

HOW WILL CLIENT CONFIDENTIALITY BE HANDLED? 

Although state agencies and SDAs may routinely share JTPA client 
Information, client confidentiality Is not an Issue as long as that 
Information Is presented In the aggregate without Individual 
Identifiers, such as client name or social security number. However, 
both the net and gross Impact evaluations Involve the merging of HIS 
data with other sources of data for which client Identifiers are 
required to accomplish the match of Information. 

To Implement an evaluation, two or more separate agencies may have to 
share JTPA data flagged with client Identifies. Each agency may have 
Its own Internal standards regarding client data access and use. For 
example, one agency may strictly limit Information containing client 
Identifiers to a small number of special users, while others may allow 
wide access to such Information. Some agencies may permit client data 
to be used for compliance Investigation and others may not. In such 
cases, Interagency discussion and agreement about client 
confidentiality must be part of the evaluation planning effort. 

As mentioned previously, assurances about client confidentiality may be 
especially Important to SDAs and service providers. Inability to 
ensure client confidentiality may Impair the cl lent-servlce-provlder 
relationship and subsequently Impact treatment success. Breaches In 
client confidentiality may also discourage ellglbles from participating 
In JTPA. For these reasons, policies regarding the use of evaluation 
data need to be established In advance. 

CONCLUSION 

Evaluation findings are only as good as the Information foundation they 
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rest on. If data are Incomplete, unreliable or Inaccessible, 
evaluation resources may he unnecessarily wasted or the evaluation 1 s 
utility substantially compromised. As a preventative measure, the 
evaluation plan should Incorporate a review of relevant data collection 
procedures and data access systems. Such a review addresses not only 
methodological concerns (data accuracy, rel lability, val 1d1ty f 
comparability) and technical concerns (data availability, computer 
capabilities), but also organizational concerns (data sharing, client 
confidentiality). In meeting these concerns, an organization Is not 
only better prepared to implement evaluation, but also enjoys certain 
long-term benefits in terms of increased data-col lection efficiency and 
accuracy affecting other oversight and research activities. 
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CHAPTER 9 
FUNDING A JTPA EVALUATION 



What Are JTPA-Related Sources of Funding? 

What are Other Sources of Funding? 
What Funding Strategy Should be Pursued? 
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CHAPTER 9. 



FUNDING A JTPA EVALUATION 

INTRODUCTION 

While JTPA legislation supports state evaluation activities, no 
specific funds are allocated to this purpose. As long as 
administrative funds remain so limited, finding financial support for 
JTPA evaluation will be a fundamental concern for most states. While 
some of the larger states which enjoy administrative economies of scale 
may feel somewhat less of a pinch, many will find current JTPA Internal 
resources already spread thin. 

The purpose of this chapter Is to encourage state JTPA planners and 
decision-makers to think broadly and creatively about funding 
possibilities for JTPA evaluation. The JTPA's orientation toward 
public-private collaboration 1n addressing employment and training 
needs sets the stage for exploring new funding partnerships 1n the 
evaluation of JTPA programs. Before examining these new partnership 
possibilities, we briefly outline various sources of support for 
evaluation within JTPA. 

WHAT ARE JTPA-RELATED SOURCES OF FUNDING? 

While legislative wording clearly supports and expects state level 
evaluation of JTPA, no monies are specifically earmarked for such 
activities. Beyond their designated five percent administrative 
monies, states may want to look at pooling administrative resources 
from other pots of JTPA money to carry out evaluation activities. Some 
obvious funds to examine are the three percent , six percent and eight 
P ercent state set-asides for special administrative and other 
activities . 
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JTPA Special Set-Asides 

In looking for sources of evaluation funding, an obvious place to start 
Is with the JTPA state set-asides designated for special 
admlnlstratalve and other activities. While use of these funds for 
evaluation may be restricted In various ways, a portion of the six 
percent, three percent and eight percent pots of money might arguably 
be applied to pertinent evaluation efforts. The evaluation-related 
possibilities for each of these set-asides are outlined here. 



• Six Percent Set-Aside : 

Much debate has already ensued around the appropriate use of six 
Percent monies for technical assistance to SDAs. The debate 
centers around what, precisely, "technical assistance" (a term 
not specifically defined In the legislation) can encompass. Is 
evaluation an acceptable form of technical assistance? In the 
past, the Department of Labor (DOL) has questioned the use of 
six percent monies for state evaluation activities because the 
legislation directs states to offer technical assistance to 
those Individual SDAs who are falling to meet performance 
standards . As of this writing, however, DOL has not taken a 
firm position, allowing states discretion on this Issue. 

Uslnj this discretion, some states have Interpreted the six 
percent more broadly to allow for evaluation. The argument here 
Is three-fold: First, states cannot adequately develop 
technical assistance packages to SDAs without first having a 
means to assess what Is or Is not effective about JTPA both 
generally and specifically at the SDA program level. Evaluation 
activities provide the necessary Information base for 
Implementing useful technical assistance. 

Second, the legislation Intended performance standards measures 
to be selective Indicators of how well JTPA programs are meeting 
certain goals, not comprehensive measures of JTPA goal 
achievement. Therefore, the purpose of technical assistance 
activities such as evaluation need not be directly and narrowly 
tied to Improving performance measures , but rather should be 
related to Improving the program's effectiveness In meeting Its 
Intended goals. 

Finally, evaluation helps spot program difficulties before they 
are reflected In performance measures, allowing for more timely 
correction of problems. Valuation therefore, may be viewed as 
"preventative" technical assistance to SDAs who might otherwise 
fall to meet standards. 

Pending a restrictive federal definition of technical assistance 
and the circumstances under which such assistance can be 
provided, states might explore the use of six percent set aside 
monies for supporting evaluation activities as a form of 
technical assistance. 
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• Eight Percent Set-Aside : 

JTPA requires that eight percent of state funds be set aside for 
state education and coordination activities. Since up to 
one-fifth of these funds may go towards enhancing coordination, 
presumably states might tap these monies to do process 
evaluations which, 1n part or 1n whole, focus on coordination 
activities . 

• Three Percent Set-Aside : 

Three percent of state administrative funds are set aside for 
special programs and services to disadvantaged older workers. 
This Is the smallest pot of state set-aside monies, and where 
program funds are funneled through the SDAs the 15 percent 
administrative restriction applies. Nonetheless, In some 
Instances, It may be feasible to use a percentage of these funds 
to evaluate special JTPA activities for older workers. 

Wagner-Peyser Funds 

The Wagner-Peyser amendments contained within JTPA (section 501) 
specifically allow for a portion of Wagner-Peyser program funds to be 
used for evaluation of Wagner-Peyser activities. Where Wagner-Peyser 
and JTPA activities Intersect, states might justify a proportionate use 
of Wagner- Peyser monies for evaluation purposes. For example, where 
JTPA evaluation calls for the manipulation of Wagner- Peyser data (I.e., 
for use In net Impact comparison group), Wagner- Peyser funds could 
contribute to that effort. Also, Wagner-Peyser provides for 
underwriting joint employment services and JTPA planning activities. 
Since evaluation Is an Integral part of planning, some of these monies 
theoretically could support evaluation of coordinated program aspects. 

Title III Funds 

Both Title III formula funding and discretionary funding allow 
significant administrative flexibility to support evaluation 
activities. In order to receive formula-allocated funds, states must 
match federal funds with their own program funds or with In-kind 
support. I-, states with greater unemployment, the match requirement Is 
proportionately reduced. While 70 percent of funds must go to direct 
service, thlr limitation applies only to federal funds and only up to 
50 percent of aj_l program funds combined. These provisions give states 
considerable latitude to Incorporate evaluation Into Title III 
activities. Evaluation costs may be counted as state match money; more 
liberal limits on administrative costs In general make support for 
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evaluation more feasible. 

Title III discretionary funds which the Secretary of Labor manages are 
to support special state training programs 1n areas of high 
unemployment, plant closures and mass layoffs. Since no state match 
money Is required and no specific legislative limitations are placed on 
these funds, states have a special opportunity to Integrate evaluation 
Into training programs sponsored by these funds. Because state program 
activities geared towards dislocated workers are a relatively new 
phenomena, the rationale for building evaluation Into these activities 
Is particularly strong. 

DOL Funds 

JTPA legislation (Sections 452-3) sets aside a portion of DOL 
administrative funds for special pi lot and demonstration projects . 
Evaluation research Is specifically named In the Act as a sanctioned 
use for this pot of money. Thus far, DOL has channeled this 
pilot/demonstration project money Into Its own research activities. 
However, this money Is theoretically available to states, as well, for 
state Initiated research projects. 

In applying this money to evaluation activities, states would have to 
carve out a particular set of evaluation research activities and 
package them within an appropriate demonstration or pilot project 
context. States would need to keep abreast of DOL research needs and 
plans In order to design projects which would satisfy both state and 
DOL evaluation Interests . Another possible avenue for obtaining 
evaluation Information would entail applying to be a state evaluation 
site In a DOL research project. The Issue In this Instance Is whether 
a participating state would have access to disaggregated data, useful 
for the state's own evaluation purposes. 

JTPA LMI Funds 

JTPA provides additional funds for developing labor market Information 
( LMI) . Since labor market factors are controlled for or considered In 
the three major evaluation approaches described 1n this guide (see 
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Chapter 2), states might rationalize using a relative portion of LHI 
funds for evaluation. 



Interdivisional Funding 

Another distinct evaluation funding strategy 1s to Involve several 
program divisions within the administrative entity responsible for JTPA 
at the state level. Such Interdivisional funding might be most 
feasible where organizational relations among divisions are fairly 
formalized, Involving frequent Interaction over common Issues. This 
pattern of Interdivisional communication creates a forum 1n which the 
case for cost-sharing 1n support of evaluation might reasonably be made. 

Such a joint funding approach might be most llkaly where JTPA 1s housed 
In the same overall administration entity responsible for employment 
services, data collection and research. In such a case, state level 
sources of evaluation funding might Include divisions like LHI, UI, 
Employment and Training, the State Employment Service and the 
Information Systems Division. 



Interagency Funding 

Similarly, decision-makers might explore funding leveraged from state 
organizations whose activities are to be coordinated with JTPA (e.g., 
Public Assistance, Vocational Education Agencies, Employment and 
Training Services, Economic Development Agencies). Trans-agency 
supported evaluation might focus more on Issues of administration and 
service coordination of Importance to the several constituent funders. 

With joint funding for a particular evaluation or for ongoing 
evaluation activities, an Independent evaluation unit attached to a 
supra-agency, such as the governor's office may be most acceptable to 
all parties. Whatever the organizational arrangement, evaluation 
activities will have to answer needs and provide recognized benefits to 
a broad spectrum of non-JTPA users. 
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WHAT ARE OTHER SOURCES OF FUNDING? 

Resources for evaluation funding exist beyond the previously mentioned 
government funding sources connected directly or Indirectly to JTPA. 
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But state and local agencies have historically been reluctant to tap 
these outside resources (and 1n the time of more munificent public 
budgets, there was perhaps little reason to explore them). Finding and 
approaching these other funders can require a staffing and time 
Investment for busy administrators that Initially discourages this kind 
of organizational risk-taking. 

Ultimately, however, casting a broader net Into funding realms beyond 
the familiar can pay off In many ways. Even 1f adventuresome searchers 
are not directly rewarded with the cash support they seek, the effort 
may still prove valuable In terms of non-monetary contributions, 
Increased contacts and Interactions within the business, academic and 
professional communities, Increased program visibility and credibility, 
and enlarged possibilities for future funding. The remainder of the 
chapter outlines some of these alternative funding possibilities. 

Universities and 4-year Colleges 

Academic Institutions can often offer unique evaluation resources at 
reduced costs. First, a major academic resource Is faculty who may 
have the specialized research expertise needed, and who are often 
available at a reduced cost compared to private consultants. Through 
their Institutional ties, faculty are sometimes better able to leverage 
related research resources ( such as research material s , computer 
expertise, other faculty and students). If the faculty consultant time 
commitment Is below a certain amount, academic Institutions will often 
reduce or waive their Indirect costs. 

Students are another potential source of support for evaluation. 
Frequently, graduate students are willing to devote research time to an 
outside evaluation project In order to gain practical work experience 
(encouraged or required by many professional graduate schools) or to 
develop material for a thesis or doctoral project. Sometimes students 
(as well as faculty) can partially or fully support their evaluation 
research activities through research asslstantshlps , post -doctoral 
fellowships or Individual research grants. While very limited, federal 
work study funds do exist at the graduate level, allowing employers to 
pay only a portion of the wage costs of a work study student. An added 



> 




ERIC 



plus 1s that students bring with them the advice, interest and support 
of supervising faculty who can act as an additional quality control on 
the student's work, and who themselves may be willing to play an active 
role in the evaluation effort, contributing specialized expertise. 

State-supported educational institutions (including community colleges) 
are also part of the state agency network. Their status provides an 
opportunity and rationale for developing closer ties that can be 
mutually beneficial to both parties. In terms of hiring a JTPA 
evaluation consultant, contracting with state-supported colleges or 
universities may be simpler, less formal, and Involve lower Indirect 
costs than would other contracting arrangements. 

for a variety of reasons, academic departments are frequently 
interested in setting up formal ties with agencies sponsoring research 
projects. Such ties might take the form of special internships for 
qualified students or reduced-fee faculty consulting. In some cases, 
graduate departments or professional schools may partially or fully 
fund studies of evaluation Issues of special relevance to their faculty 
and students. One JTPA evaluation, for Instance, was largely sponsored 
by a local university's graduate business school. Faculty and SDA 
staff planned the evaluation; students collected and analyzed data 
under faculty supervision. When collaboration with a university is 
more formalized, faculty are more likely to play an active role in 
screening and supervising students. 

Special Organizations 

A number of non-profit business, labor, professional, social service 
and public Interest organizations have a special Interest in evaluating 
and thereby Improving employment and training programs. A 
state-sponsored JTPA evaluation may be able to capitalize on this 
Interest in a number of ways. For example, members of such groups 
might act as formal or Informal advisors to the evaluation planning 
process. Members might be willing to offer reduced fee services or 
provide certain resources in exchange for public recognition of their 
contributions. The National Alliance of Business (NAB), for example, 
has contributed to local JTPA evaluation activities. Other 
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organizations might also be willing to lend various forms of support. 

Private Foundations, Charitable Organizations and Trusts 
Private foundation support used to be almost entirely the preserve of 
educational Institutions and non-profit organizations . Increasingly, 
however, public agencies have broadened their funding strategies to 
Include soliciting foundations for support. Nor 1s foundation support 
limited to direct services; many foundations are concerned with 
developing Innovative approaches to service delivery and are willing to 
fund applied research activities (such as evaluation) 1n a number of 
service areas, Including employment and training. 

Host major metropolitan libraries carry standard directories (refer to 
the reference section for examples of these directories) profiling the 
larger national and regional foundations and their giving patterns. 
Regional directories of state and local funders are also usual ly 
available. Such directories provide Initial Information needed to 
Identify those funders who are most likely to be Interested 1n social 
program evaluation activities and 1n employment and training Issues. 

The major directories Include fairly detailed and historical profiles 
on foundation activities (previous finding patterns, kinds of costs 
covered, special requirements , current recipients of support) , which 
help the researcher quickly narrow the search effort. Financial 
r eports of foundations, charities, and trusts within a state also give 
a good sense of who and what these organizations fund, their funding 
philosophy and agenda. (These reports are generally available through 
the state attorney general 1 s office or the state agency which oversees 
the financial reporting of charitable organizations.) 

These funders may be more attracted to programs which are Innovative or 
can serve as demonstration models for other programs. Evaluation of 
programs geared to special p opulations (e.g. , youth, ex- of fenders , 
welfare recipients, older workers) may also resonate with certain 
funders who otherwise would not be Involved with JTPA evaluation 
activities. 
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Size and location of foundations are often Important considerations. 
Smaller and more local foundations may be more unpredictable 1n their 
outlook, They may support an especially appealing project outside 
their usual framework. In contrast, the larger national foundations 
are more bureaucratic, engage In a very formalized selection of Issues 
to be funded, have more specifically defined application procedures and 
fixed funding parameters, and apply more rigid criteria 1n making 
funding decisions . Larger foundations also tend to have lengthy 
timeframes for review and final deds 1on-mak1ng. The trade-off Is that 
major foundation support, while more competitively sought, more 
difficult and time-consuming to achieve, offers larger pots of money, 
greater prestige and Increased 1 Ikel ihood of supplemental funding 1n 
the future. 

Private Business sector 

JTPA envisions a close working relation between government and the 
private sector to better connect those who are being trained with those 
•who can offer jobs. In the Interest of learning more about and 
Improving current JTPA operations, the public-private partnership might 
arguably be extended to Include joint support for evaluation activities. 

Large companies utilizing JTPA services such as OJT may be particularly 
receptive to requests for assistance In evaluating and Improving those 
services, (rtore support may be available If the company also views Its 
participation In terms of public relations returns.) While state 
agencies may be unaccustomed to approaching the private sector directly 
for help, a mechanism for making such contacts Is built Into the JTPA 
State Councils and PICs. The project which sponsored this set of 
evaluation guidebooks Is a prime example of how private businesses may 
join with the public sector In supporting evaluation activities. 

In addition to approaching business contacts through JTPA channels, 
other sources of Information on private sector companies are available 
to help 1n the fund search. State employment agencies, economic 
development organizations and private research companies often publish 
Information on the largest employers 1n the state. Also, major 
university and public 1 Ibrarles usual ly carry reference guides on 
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corporations 1n each state which describe their giving programs. (See 
the references section for specific references.) 



Local companies can be contacted directly for Information about their 
funding Interests and requirements . Usually, the funding proposals do 
not need to be as long or as complex as with other funders and the 
decision time Is much shorter. With major national corporations, the 
scenario can be quite different. They often have special (usually 
non-local) corporate giving units that handle all funding requests, 
often requl ring somewhat more sophl stlcated and detal led proposals . 
While these special units may make the final selections, local 
corporate branches may also wish to be Involved In the review process 
and may have Influence over the ultimate funding decision of corporate 
headquarters. 



WHAT FUNDING STRATEGY SHOULD BE PURSUED? 

Funding sources of all kinds have reduced their giving programs over 
the past few years. Creative, Imaginative and well thought-out funding 
strategies have always made a difference, but now they are Imperative. 
In the present period of scarcity and shifting social welfare values, 
funding social services Is a genuine challenge. It is also 
Increasingly difficult to locate funders with a special Interest In the 
assessment of employment and training efforts. Therefore, the fund 
searcher must build maximum efficiency Into the fund search effort. 
Following are some strategies for developing JTPA funding proposals and 
Increasing the likelihood of their success. 



1. Identify potential funders of policy research, 
particularly ongoing program evaluation or the 
evaluation of pilot and demonstration programs in the 
human services. 

The economics and business sections of most public libraries have 
excellent directories on foundations and corporations. Repositories 
for government documents In colleges, universities, and state libraries 
have Information on government funders . The G rants and Contracts 
Weekly and The Business and Commerce Dally are the most current sources 
of Information on government funding priorities. Automated searches 
provide quick sources of Information on a range of private and public 
sector funders . A large pool of "possible" funders can therefore be 
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WHO SHOULD DO THE EVALUATION? 



3. Develop a General Fund Search Strategy. 

Work plans and timeframes are generic to government. They are just as 
necessary for fund search as for the development of the projects for 
which outside funding 1s sought. Although In the final analysis, such 
plans and timetables must be tailored to different funders, It Is 
helpful to begin with an overall strategy which Is modifiable. Such a 
stategy has at least the following elements: 



• The preparation and/or acquisition of basic fund search 
materials 

• The preparation of a general description of the project to be 
funded 

• The securing of general letters of support for the project, 
from Individuals whose endorsement will likely Increase the 
credibility of the funding request. 

• The preparation of a general cover letter to accompany these 
materials 

• The development of a chronological work plan and timeframe for 
obtaining funds, based on considerations of staff resources, 
time pressures, the need to acquire funds from more than one 
source, the ability to maintain organizational support from the 
project sponsor ever time, the realities of governmental and 
nongovernmental funding cycles , and other organizational and 
political considerations. 

4. Fine-tune the iaaterials to each of a small set of top 
priority funders within the "likely" group. 

The most Important aspect of tailoring a funding request Is to achieve 

an honest mesh between the characteristics of the proposed project and 

the current priorities of the funder (and to a lesser extent the 

funder's historical giving pattern). The goal Is to construct an 

Individualized funding rationale for each potential funder to be 

approached. 



5. Decide which is more appropriate: A single-f under or 
roulti-funder approach. 

If the latter, each funder should be Informed 1n the cover letter what 

others are being simultaneously approached. Learning this from other 

funders 1s often the kiss of death for a funding application. A 

staggered approach to multiple funders may 1n some cases be the best 

method. Securing one major funder may tend to leverage funds from the 

others. Corporate givers may be resistant to being the only private 
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funder; a multl -f under strategy :hould consider Incorporating more 
than one private funding proposal. 

6. Update critical information on the lunder(s) selected 
for the first phase of the fund search. 

You may need to solicit fresh Information on funders through personal 

contact with an Individual within the funding organization who Is In a 

position to give you the Information you need relative to the kind of 

project being proposed. It Is wise, however, not to Identify the 

project or Its sponsor at this point, since this can affect funding 

decisions prematurely. Rather, this should be a general 

Information-gathering call, confined to questions such as: 

• What are the funding application guidelines and procedures? 

• Where and to whom does one submit an application for funds? 
(Application materials should be requested, since many funders 
require a high level of conformance with their formal 
procedures.) 

• What are the current funding priorities? (Some corporations 
and smaller foundations will not tell $ou. The larger 
foundations have elaborate booklets outlining and justifying 
their current areas of Interest.) 

• How flexible Is the application process? 

• What additional factors may feed Into the selection process? 

• Are public sector programs likely to be considered seriously 
for awards? 

• Who Is the best contact person for following up on the status 
of an application? 

• What other kinds of things will the funder look for in an 
application? 

In studying these specific characteristics of funders, searchers n1ll 
then be better able to furl her narrow the fund search to a few select 
and optimal choices. 



7. Identify special internal resources and capabilities. 
As part of a well-crafted proposal, a fund-seeker will want to 
emphasize those specific organizational resources and capabilities 
which will positively affect the evaluation process. Funders will be 
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looking for special characteristics that set the fund-seeking 
organization apart, characteristics that suggest the organization will 
be able to carry out the proposal 1n a successful, effective manner. 

For example, many funders are Impressed with proposals that appear to 
marshal effective community support or Involvement or that have already 
obtained contingency funding from other sources . Al so, some funders 
may favorably view projects for which consultants or In-house staff 
with requisite training and experience for the project have already 
been Identified by fund-seekers. 

8. Develop cooperative relations with organizations which 
can act as funding brokers. 

Private sector funders at any level are likely to be resistant to 

funnellng support directly to public sector agencies or local 

governmental units. JTPA fund searchers will therefore, want to 

explore the use of "funding brokers 11 for their proposals. Such brokers 

might Include relevant university departments, research Institutes, or 

an appropriate non-profit organization which agrees to pass through the 

funds to the state agency. In exchange, the broker may expect some 

level of participation In the project or may charge for Indirect costs 

In acting as a funding conduit. 

9. Solicit powerful advocates who can call or write to the 
funder on behalf of the project at an appropriate point 
in the review process. 

This Is a sensitive Issue which must be carefully handled and timed. 

Too much and too little advocacy can be a problem. States may have few 

personal contacts within the private sector funding world and have to 

rely on advocacy support from within state government (e.g. , the 

governor, legislators, congressional representatives) . 

10. Submit funding applications to the preferred funders, 
followed by a call to contacts within the funding 
organization to check on their receipt of the 
application and to clarify the review and selection 
process . 

Applications can take the form of finely crafted cover letters 
accompanying long proposals conforming to a myriad of strict 
guidelines, or they can Involve brief cover letters oriented to the 
funder 1 s primary funding purpQses accompanying a short, concise concept 
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paper cn the project. Whatever the required format, the rationale for 
a particular funder to support a project must be clearly presented. 



11. Wait patiently for an acknowledgement that your 
application has been received and for most of the 
review process to have taken place, and then implement 
an advocacy effort, 

Most large foundations politely notify the applicant and keep the fund- 
searcher Informed about the process . Host corporations do not. 
Smaller foundations and employers are often very amenable to calls from 
applicants . 

The source of advocacy Is Important. Pressure from elected officials 
may work well with government agencies but not necessarily with private 
foundations or corporations. Local foundations are affected by 
advocacy from the client groups Involved, or from client advocacy 
groups In the community. 

12. If the first wave of fund search activities fails, 
select another set from the "2 ikely" pool and begin 



You will want to seek Information on why the first choices turned you 
down. This may help you revise your concept papers and proposals, as 
you tailor them for new funders. In gearing up for another round, you 
may want to consider a different kind of funder- smaller, or closer to 
the project, or go the other direction. You may prefer a 
private/public partnership strategy this time, If you tried for a 
single funder the first time. Skill, Imagination, flexibility, 
patience, and confidence In you project are essential In modifying your 
general funding strategy to accomodate for the normal series of wins 
and losses In fund search. 

CONCLUSION 

For many states Interested In evaluating JTPA, funding will be an 
Important preliminary hurdle to negotiate. While new evaluation 
responsibilities have fallen on states, traditional government funding 
sources under JTPA are far more limited than 1n the days of CETA. As a 
result, funding strategies may have to rest more on combining financial 
and 1n-k1nd support from several funding sources. Various JTPA-related 
administrative pots of money are obvious sources for partial funding of 
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limited JTPA evaluation activities. For some states, joint funding 
arrangements within or across JTPA-1nvol ved agencies may prove the most 
feasible way to sustain an ongoing evaluation capability. 

Non- traditional funding sources should not be overlooked. Public 
sector administrators, generally unaccustomed to venturing beyond 
government funding options, win clearly have to move towards engaging 
support outside as well as Inside the public sector. Universities, 
professional and community organizations, business and labor groups, 
private foundations and corporations may represent Important untapped 
resources for carrying out JTPA evaluation. 
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CHAPTER 10 
STAFFING A JTPA EVALUATION 



Will Evaluation Require Special Staffing? 

Who Should Staff an Evaluation? 

The In-House Approach 
The Outside Consultant Strategy 
Compromise Strategies 

Consultants: What Are the Options? 

Finding and Selecting a Consultant 
Contract Concerns 
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CHAPTER 10. 



STAFFING A JTPA EVALUATION 



INTRODUCTION 

Because each state will have Its own evaluation Interests and needs, 
every evaluation effort will be somewhat unique; no single staffing 
pattern suffices for all. In some settings, an In-house team of 
evaluation specialists Is most feasible; In other contexts, an outside 
consultant may make more sense. Each approach has potential advantages 
and disadvantages which will be outlined later In this chapter. An 
Important consideration Is whether available 1n-h..jse staff have the 
technical skills to accomplish the kinds of evaluation tasks that are 
required, in addressing this consideration, we look first at some of 
the specialized staffing needs an evaluation might entail. 

WILL EVALUATION REQUIRE SPECIAL STAFFING? 

Comprehensive evaluations will likely require evaluation specialists In 
areas such as research design and statistical analysis; more scaled 
down efforts might manage with fewer expert resources acting In a more 
limited consultant fashion. Whatever the scale, most evaluations will 
require some special staffing. The charts which follow present a rough 
notion of the sorts of special staffing needs an evaluation might 
engender: 
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CORE EVALUATION STAFF 



Type Of 
Specialist 



Examples of 
Specialist Activities 



Program Evaluator 
(specializing In employment 
and training programs) 



Develops and Implements a feasible 
overall evaluation approach (the 
questions to be Investigated) and 
methodology to meet the Information 
needs of a state or SDA. 



Coordinator of Evaluation 
Activities 



Coordinates activities In support of 
evaluation. Assesses the supports and 
constraints for conducting evaluation; 
develops strategies for increasing the 
utility and utilization of evaluation. 
Coordinates activities across agency 
and division boundaries. P.ans and/or 
coordinates resource utilization, 
staffing, and other implementation 
components of the evaluation. 



HIS Programmer/ 
Analyst 



Develops programs needed for merging 
categorical data from different sources 
Creates customized data sets for 
analysis purposes and does data 
analysis under the supervision of the 
program evaluator. 



Surveyor, Interviewer or 
other Data Collectors 



Carries out the actual collection of 
Information required by the evaluation 
research approach. 
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ADDITIONAL EVALUATION SPECIALISTS 



Type of 
Specialist 



Evaluation Researcher 
(specializing In 
evaluablllty assessment) 



Examples of 
of Specialist Activities 



Determines the feasibility of carrying 
out different kinds of program eval- 
uations, given a state or SDA's 
evaluation needs. 



Research Design Specialist 



Advises a program evaluator on the most 
appropriate and efficient strategies 
for data collection and analysis. 



Sampling Specialist 



Advises program evaluator on sampling 
strategies to ensure maximum validity 
and reliability of Information 
collected. 



Survey Researcher 



Advises on the construction of 
Interviews and questionnaires. Assists 
In Implementation of phone, mall, or 1n- 
person surveys of JTPA participants, 
employers and others. Trains and super- 
vises Interviewers. 



Applied Social Statistician 



Advises on appropriate and efficient 
methods for statistical analysis of data 
In order to obtain valid Information. 



Public Information Staffer 



Assists 1n promotion of evaluation 
effort, developing Informational 
materials and/or funding solicitations. 
Assists 1n packaging and dissemination 
of final reports. 
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At first glance, the above 11st of specialized staffing needs may seem 
formidable. However, the 11st 1s offered not to discourage, but to 
realistically present some of the distinct resources evaluation will 
have to draw on In order to provide truly useful Information about 
JTPA. The experts listed In the second chart (Additional Evaluation 
Specialists) are necessary only If the evaluation questions to be 
answered present particular research challenges where the core staff 
must turn for special advice. Moreover, a small core research staff 
can encompass a number of these skills so that staffing costs need not 
be prohibitive. One state for example, accomplishes much of Its 
ongoing JTPA evaluation work with one research dl rector and two 
assistants. 

WHO SHOULD STAFF AN EVALUATION? 

Two major staffing configurations for carrying out evaluation are 
possible: 1n~house staffing and outside consultant staffing. Each has 
Its decided pluses and minuses, which will be more or less pronouced 
depending on the particular evaluation context. The following 
discussion touches on the potential advantages and disadvantages of 
each staffing approach and offers some compromise strategies combining 
both. We begin with an examination of the In-house staffing approach. 

The In-House Approach 

Some states are meeting the JTPA evaluation challenge through creative 
In-house approaches. While many states never have themselves conducted 
comprehensive evaluation of their employment and training programs, 
they often have access to untapped resources sufficient for such an 
undertaking. These resources may be drawn from several agencies or 
several divisions within a lead agency and brought together under one 
roof as a special team, or loosely coordinated as a consultant panel. 

Certainly, cost Is one of the most compelling arguments for seeking 
1n-house expertise. However, In certain settings, such an approach may 
Involve many hidden costs which need to be entered Into the overall 
calculation In deciding which staffing strategy to pursue. 

First, 1t may take significant time and effort to locate and engage 
special evaluation staff. Division or agency heads are likely to be 



skeptical and resistant to loaning personnel (underscoring what has 
been said earlier about the importance of building broad organizational 
support for evaluation) . Also, pooling 1n-house staff resources may 
require extra management staff to bridge the communication and 
coordination gaps that Inevitably will arise. And finally, there may 
be some Inefficiencies associated with less experienced and less 
specialized staff attempting to progress along a learning curve while 
evaluating JTPA. 

Cutting corners on evaluation special Ists may ultimately cost the 
organization far more than would have originally been spent on 
consultant fees. Where in-house evaluation staff lack requisite 
technical expertise, the great risk is that the Information obtained 
will lack sufficient reliability or validity; the findings will be of 
diminished value. A less obvious cost of using in-house evaluators may 
be lower credibility of the evaluation results. 

However, the in-house approach to evaluation also carries some less 
obvious, but potentially Important benefits, which include: 

• Evaluation staffs familiarity with the organization setting, 
data collection systems, staff capabilities, time schedules, 
program procedures, etc.. 

• Fewer entry problems for evaluation staff, more rapport with 
program staff; greater receptivity to programmatic needs of 
stafF. 

• Cost savings potential through closer monitoring and control of 
the work in progress. 

• Opportunities to foster Inter- and Intraagency communication. 

• Capacity-building for further evaluation efforts. 

• Flexibility in reassigning evaluation staff to evolving tasks. 

In-house staff may also provide continuity to the evaluation process. 
Staff are present at the beginning, so that evaluation needs are 
accomodated In program design and evaluation; staff are also present 
after the evaluation, to fadl itate and encourage the programmatic 
changes identified as useful. 
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Building an In-House Evaluation Capability 

In building a JTPA evaluation capabll 1ty f states have a number of 
options. Given the wide range of evaluation needs, tn-house 
capabilities, and organizational constraints In each state, no one 
option can claim clear superiority. The staffing approach that Is 
effective In one setting, may be Ineffective In another. Of particular 
concern In assessing the appropriateness of a staffing strategy are the 
six criteria mentioned In the Chapter 4 discussion of where to locate 
an evaluation unit. Again, those criteria are: 

• Position within the authority structure 

• Separation from compliance functions 

• Neutrality 

• Trust 

• Coordination capabilities 

• General competency 

When applied to different staffing approaches, these criteria suggest 
pluses and minuses and distinct trade-offs between those approaches. 
Each state will have to judge for Itself how It may best develop Its 
evaluation capabilities, given the organizational framework within 
which Its JTPA programs operate. For a specific checklist of concerns 
about who should do evaluation, see the following page. 

The Outside Consultant strategy 

Within the evaluation community the debate over whether to use In-house 
resources has been ongoing. Obviously, In circumstances where access 
to In-house expertise Is limited, turning to outside evaluation 
specialists Is the only option. 

However, critics of the In- house approach argue that even If In-house 
resources are available, some Important potential benefits offered by 
outside consultants should not be overlooked. These potential benefits 
Include: 
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WHO SHOULD DO THE EVALUATION? 



AUTHORITY FACTORS YES NO NOT SURE 

1. Will evaluation staff be removed from the 

organizational hierarchy of programs being evaluated? |_J |_~| |~| 

2. Will evaluation staff report directly to key 

decision-makers? I j | | | — | 

3. Will evaluation activities be separate from 

compliance activities? | i i i ■ — ■ 

4. Will evaluation staff have sufficient status to _ _ 
obtain necessary cooperation with program staff? | | | | j~| 

INDEPENDENCE AND NEUTRALITY FACTORS 

1. Will evaluators 1 organizational status permit their _ 

Independent judgment and action where appropriate? | | | | | | 

2. Will those being evaluated receive evaluators as 

Independent and neutral? | | i i i j 

3. Will funders or other decision-makers view 

evaluators as Independent and credible researchers? | | |_J | | 

TRUST FACTORS 

1. Will evaluation staff have requisite Interpersonal 

skills? j— | j — j 

2. Will evaluation staff have good rapport with program _ 

staff and ready access to Information? | | | | | j 

3. Will decision-makers be likely to trust the 

evaluation findings? | | j | | | 

COMPETENCY AND COORDINATION FACTORS 

1. Will evaluation staff have requisite skills/ 

expertise? I i I I I I 

2. Will evaluation staff Include those with specific _ _ _ 
experience In evaluating employment and training | | |_| | | 
programs? 

3. Will evaluation staff be familiar with the JTPA 

system? I ~| | | j — | 

4. Will evaluation staff Include those with good _ _ 
organizational, planning and management skills? |^J | | |_| 

5. Will evaluation staff be able to effectively use _ 

and develop communication and coordination channels | | | | I I 

among JTPA actors? 
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• Greater credibility with evaluation users, particularly funders. 

• Separation from the organization which allows for greater 
objectivity and fairness (actual or perceived). 

• More acceptance from program staff who feel less threatened. 

• Greater assurances of a quality product produced by an 
experienced specialist. 

• Greater cost effectiveness In the long run. 

• Ability to allow staffing levels to fluctuate In response to 
varying resource needs. 

Outside evaluations may be most appropriate In situations where 
organizational tensions or mistrust call for an evaluation with maximum 
separation from the JTPA system. One state, In evaluating SUA 
activities, plans to team a state evaluator with a private consultant to 
reinforce a sense of Independence and neutrality In this sensitive 
undertaking, yet still be able to take advantage of the knowledge of JTPA 
that an In-house evaluator offers. 

Compromise Staffing Strategies 

A compromise staffing strategy Involves the judicious use of consultants 
at critical planning and Implementation junctures of the evaluation where 
expertise Is most needed. For example, a consultant might be brought In 
solely to assess the evaluablllty of a program (see Chapter 3) or to 
develop the evaluation design which others may carry out. Alternatively, 
a consultant's role might be strictly advisory, limited to reviewing and 
commenting on the In-house evaluation work In progress. In this manner, 
quality control might be assured, while consultant's fees are contained. 
When a formal review Is conducted by a completely Independent party, the 
process may be considered an evaluation audit , as described below. 

Incorporating Audit Procedures into the Evaluation 
An In-between staffing solution 1s to supplement 1n-house evaluation 
activities with external auditing of those activities. In essence, the 
audit constitutes an evaluation of the evaluation, a process sometimes 
referred to as M meta-evaluat1on. " 
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This kind of audit by an Independent third party serves several 
functions. An auditor can formally review and critique not only the 
evaluation plan, but also Implementation procedures and the final 
evaluation report. By reviewing the plan before evaluation commences, 
the auditor can spot problems, gaps and weaknesses In the plan and 
suggest changes to Improve the scientific soundness, the organizational 
effectiveness, or the efficiency of the evaluation. Using an outside 
auditor not only can Improve the utility and appropriateness of the 
evaluation, but also car enhance the credibility of an effort planned and 
executed by In-house staff. Because using an auditor offers many of the 
protections of contracting out an evaluation but at a much reduced cost, 
It 1s an attractive staffing alternative. 

Audit Criteria 

If an auditor Is to be used, his or her contract should specify, among 
other things, the timing and manner in which the audit will be carried 
out, the evaluation elements to be examined and how findings will be 
presented. (For more on selecting a consultant, see the last section of 
this chapter.) The specific criteria for evaluating an evaluation will 
obviously vary with the Individual setting, but need not be confined only 
to considerations of research approach and methodology. The evaluation's 
soundness may also be judged In terms of its organizational 
appropriateness, utility and cost-effectiveness. On the following page, 
a checklist of meta-evaluatlon criteria adapted from StufMebeam (1974) 
Illustrates the critical breadth an evaluation audit may entail, 

FINDING CONSULTANTS: WHAT ARE THE OPTIONS? 

Choice of consultants Is not limited to the fv»w listings ' d md In the 
Yellow Pages directory, or to RFP respondents. However, indlng other 
consultant options will entail some Initial effort in stepping outside 
familiar agency territory to ferret out new Institutional contacts both 
in the public and private sector. 

Finding a Consultant 

While options for outside assistance will be different In each state, the 
list below summarizes some of the basic kinds of external resources 
available to a JTPA evaluation development effort. 
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CRITERIA FOR AUDITING AN EVALUATION 



CRITERIA FOR TECHNICAL ADEQUACY 

• Internal validity - whether the findings are accurate 

• External validity - the extent to which the Information Is 

generallzable" (I.e., the range of persons and conditions to 
which the findings can be applied) 

• Reliability - whether the data are accurate 

• Objectivity - whether the data are likely to be Interpreted 
similarly by different competent judges 

CRITERIA FOR UTILITY 

• Relevance - whether the findings relate to the purposes of the 
program 

• Importance - whether the evaluation covers the most essential 
features of the program 

• Scope - whether the evaluation addresses all of the Important 
questions 

• Credibility - whether the audience trusts the evaluators and 
supposes them to be free of bias 1n conducting the evaluation 

• Timeliness - whether the evaluation findings are available In 
tliiie to be used In making decisions 

• Pervasiveness - whether the findings are disseminated to all 
Intended audiences 

CRITERION FOR COST-EFFECTIVE&ESS 

• C^st-effectlveness - whether the evaluation costs are kept as low 
as possible without sacrificing quality 



From: Daniel L. Stuff lebeam, "Heta-Evaluatlon . " Occasional Paper No 3 
Kalamazoo, HI: Evaluation Center, Western Michigan University, 1974. 
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• Universiti es and Colleges : Both faculty and students may have 
specialized expertise they would like to lend to an applied 
evaluation research setting. While academic Institutions 
rarely have a specialized degree program 1n evaluation, many 
departments, such as business administration, planning, public 
affairs, economics, sociology, political science, and social 
work will house Individuals with an expertise In evaluation 
research. Not only faculty, but graduate students under 
faculty supervision might be able to offer valuable expertise. 
A possible constraint to using faculty and graduate students Is 
the limited time they might have to devote to outside 
consulting and research. On the other hand, faculty are often 
better trained for specialized evaluation requirements and are 
often less costly as a staffing alternative. 

" Research Institutes : Even If the research Institute Itself 
does not have appropriate specialists, Institute personnel may 
be plugged Into a broader network of researchers which Include 
the right kinds of specialists for a particular JTPA evaluation 
effort. 

• Professional Groups : Evaluation research encompasses a number 
of professional associations. Organizations such as the 
Evaluation Research Society (a national professional 
association for evaluators) or the American Sociological 
Association can be of use In locating qualified evaluators 
within a given area. Some states are also actively tapping 
such associations for assistance In doing JIPA evaluation 
planning. 

• SDA's and Local Governme nt: Because of their history working 
with CETA and other training and development programs, SDA and 
city or county planning staff may offer Important perspectives 
on available consultants. 

" Business and Labor-Affiliated Organizations : Many such 

organizations are also keenly Interested In JTPA and may have 
staff or other contacts Interested In participating In an 
evaluation effort. The National Alliance for Business (NAB), 
for example, has been directly Involved In the staffing of 
local JTPA evaluations. The labor-affiliated Human Resources 
Development Institute (HRDI) has also been active In JTPA 
planning and assessment Issues, particularly In the Title III 
programs. 

In exploring any of these options, the key Is developing ongoing 
contacts within the network of researchers affiliated with these groups 
to maximize the chances of finding the right kind of consultants at the 
right price. Many times, consultant resources through these groups are 
available at a much reduced cost or additional organizational resources 
are at the consultants ' s command. (Refer to Chapter 9, "Funding a JIPA 
Evaluation," for more on utilizing outside resources to support 
evaluation. ) 
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Selecting a Consultant 

Consultants 1 fees vary tremendously and so do the quality and types of 
services offered. There 1s no- fool proof method for guaranteeing an 
appropriate and quality product from a hired consultant (although 
controlling the purse strings helps). However, some preliminary 
assessment (even though It may be 1rr1tat1ngly time-consuming) of the 
consultant and the consultant services offered wl 1 1 Increase the 
chances of choosing wisely. Preliminary assessment might Involve: 

• Reviewing the consultants resume and written products 

0 As a first requirement, does the consultant have the 
requisite specialized research skills and training 
necessary to carry out the particular activities needed? 

° Do the products have clarity and depth? 

° Are materials well-written, understandable? 

0 Do products suggest the consultant has skills and 
experience applicable to the task at hand? 

• Interviewing the consultant 

0 What are the consultant's areas of expertise and training? 

0 What are his or her conceptual or methodological biases? 

0 Does his or her approach to evaluation fit with your 
particular program's needs? 

0 How sensitive Is the e valuator to organizational factors 
affecting evaluation? 

° How will the evaluator fit In? How well will he or she 
relate to others on the evaluation team or In the program? 
How Independent will he or she be? 
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• Contacting previous contractors 

° How timely have previous efforts been? 

° What Is the quality of previous work? 

• How well did the contractor work with others? 

° Were any problems or difficulties encountered? 

• Requesting a written plan of action (Works best If 
evaluation priorities have already been established and the 
role of the evaluator within the overall framework of the 
evaluation Is fairly well-defined.) 

° How well does the evaluator grasp his or her role? 

How creatively does the evaluator deal with limitations and 
constraints? 

0 What special resources can the evaluator marshal from 
outside? (e.g., access to computer use, word processing, 
other consultants.) 

Contract Concerns 

The final step In selecting a consultant Is Ironing out a contract that 
both parties will be satisfied with. A good contract anticipates areas 
of potential ambiguity or conflict and protects both the consultant and 
the contracting agency. Among other things, the contract should: 



Specify all interim and final products and a timetable for each 
product's completion. Requirements for Interim products are 
especially important In a large or lengthy project to keep the 
project on track and to allow for review, comment and revision. 

Detail specific roles, responsibilities, lines of authority and 
decision-making procedures in the evaluation project. 

Define which resources (such as secretarial and other staff, 
computer time and copying machines) the evaluator will have 
access to, and In what ways such access will be delimited. 

Include any follow-up responsibl 1 1 ties the consultant might 
have once the evaluation is complete, such as making in-person 
presentations of the findings to specified groups. 

Determine what proprietary rights the consultant has In the 
evaluation findings or products. 

Determine what kind of confidentiality requirements the 
consultant must agree to observe. 
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• Hake payment conditional on satisfactory Interim and final 
products, specifying (as clearly as 1s possible) what 
constitutes "satisfactory 11 and through what process the 
acceptability of a product will be determined. 

• Outline expectations and a timetable for revision work. 

• Include a termination clause allowing either party to terminate 
the contract with proper advance written notice. 

CONCLUSION 

There Is no magic formula for staffing an evaluation effort. In 
choosing a particular staffing configuration, so many factors enter in: 
the level of 1n-house talent and expertise, staff availability, 
comparative costs of different staff choices, credibility factors and 
other political considerations, to mention a few. As the last sections 
Imply, finding and selecting a consultant to complement evaluation 
activities tacks on additional time costs in interviewing, assessing 
consultant products and past performance, and assembling and 
negotiating a contract. Given all these staffing considerations and 
concerns, each state must determine what evaluation staffing pattern is 
most efficient and feasible. Hopefully this chapter has offered some 
useful guidelines in making this determination. 



ERLC 



148 



149 



VOLUME CONCLUSION 



This guide has focused on various planning and Implementation Issues 
which will likely confront a state-level JTPA evaluation effort. A 
primary goal has been to help JTPA practitioners anticipate the kinds 
of planning and resource commitments an evaluation might entail. A 
theme which threads throughout the various planning steps described Is 
the Importance of the organizational context to the evaluation 
process. Initial planning effort must be devoted to assessing the 
organizational supports and constraints to evaluation and developing 
evaluation strategies which are responsive to this organizational 
framework. Various stakeholders within JTPA must be brought Into the 
planning process early to nurture their Involvement and commitment tc 
the undertaking and tc Insure greater relevance and utility of the 
evaluation findings. In the planning stages, evaluation staff may play 
a key role In bringing together diverse actors within JTPA and creating 
new patterns of communication and cooperation. 

Wnlle pointing out potential Issues and problems areas, this guide's 
central message to JIPA practitioners Is one of encouragement In the 
evaluation undertaking. Evaluation can make a difference to state JTPA 
managers and policy-makers needing specialized Information about how 
efficiently and effectively JTPA goals are being met. And evaluation 
can result In indirect organizational benefits, such as enhanced 
credibility, Improved organizational structure, or more efficient and 
accurate data collection. 

Before committing to evaluation, JTPA decision-makers not only want to 
be certain of Its returns; they also need to know that the entity will 
have the capabilities for successfully carrying out the endeavor. For 
this reason, the guide has given added emphasis to specific 
Implementation concerns. An underlying premise throughout is that 
state JTPA organizations, despite Internal JTPA funding restrictions, 
have a number of options open to them In organizing, staffing, and 
funding an evaluation. In exercising these options, JTPA staff may 
make valuable new connections with other governmental agencies, 
universities and colleges, and private sector organizations. 
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Appendix 

A Report on a National/State Survey of 
Local JTPA Constituencies 



By Bonnie Snedeker 

Snedeker Scientific, Inc. 
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INTRODUCTION 



This report summarizes the findings of a telephone 
survey on the JTPA evaluation priorities and 
capabilities of local-level constituencies. The survey 
was conducted by Snedeker Scientific, Inc. and the 
Seattle-King County Private Industry Council during 
February through April, 1985. Included in the survey 
were SDA, PIC, and local government representatives 
from 24 SDAs - 12 in Washington State and 12 from a 
specially selected national sample. 

The survey is part of the JTPA Evaluation Design 
Project, conducted by the Washington State Employment 
Security Department, with support from the National 
Commission on Employment Policy and the I.B.M. 
Corporation. This project , through the combined 
efforts of a team of evaluation design specialists and 
a national advisory committee of state and local 
practitioners, will develop and produce a series of 
evaluation models which can b£ used to assess the 
effectiveness and impact of JTPA programs and systems 
at the state and SDA levels. 

The purpose of the survey is to provide input from 
local constituencies which can help the evaluation 
designers to develop models appropriate for JTPA 
evaluation at the local as well as the state level. 
Specifically, the survey sought useable information in 
the following areas: 1) the climate for evaluation 
initiatives at the SDA level; 2) local priorities and 
needs for the use of evaluation information; 3) local 
issues and priorities in regard to specific evaluation 
measures and design capabilities; 4) local suggestions 
for evaluation designers; and 5) local system 
capabilities and contingencies. 

The survey was designed to produce maximum input from 
local constituencies within some fairly tight resource 
and time constraints. We make no claims of statistical 
significance for its findings. It is best viewed as a 
practical research effort, intended to tap local 
constituencies on a selective basis and produce 
descriptive and qualitative information of particular 
interest and utility for evaluation designers. It is 
hoped that these findings will be of interest also to 
those who participated in the survey, as well as others 
concerned with JTPA . evaluation at the SDA level. 




The report is organized into seven major sections. The 
rirst section provides an overview of the methodology 
employed in designing and conducting the survey and 
analyzing survey responses. 

The second section summarizes our analysis of the local 
climate for JTPA evaluation and the potential 
receptivity of local constituencies for evaluation 
models and guides. 

The third section of the report focuses on how 
evaluation information is used, or could potentially be 
be most useful, at the SDA level. It describes in some 
detail local uses and needs for evaluation information 
in five priority areas: 1) performance management; 2) 
local policy development; 3) program funding, design 
and development; 4) publicly documenting 

accomplishments; and 5) marketing. 

Section four lists and summarizes local constituency 
priorities for various measures of longer-term outcomes 
and effectiveness. 

Section five lists and comments on the evaluation 
capabilities rated by local constituencies as priority 
elements for inclusion in an overall evaluation design 
package. 5 

The sixth section "summarizes respondents' suggestions 
for consideration that evaluation designers should bear 
in mind in developing and producing JTPA evaluation 
models and guides for use at the SDA level. 

The final section of the report examines current local 
system capabilities and potential constraints in four 
areas: i; data collection, retrieval, and analysis; 2) 
funding support; 3) staffing support; and 4) PIC 
support. 
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METHODOLOGY 



The survey approach was designed by Bonnie Snedeker 
(Snedeker Scientific Inc.) and Brian 0 1 Sullivan 
(Seattle-King County PIC) under the direction of the 
Project Coordinator, Ann Blalock (Washington State 
Employment Security Department.) 

It was determined at the outset that, given the time 
and financial resources available, the best approach 
for tapping useful input from local constituencies 
would be a telephone survey of a limited but carefully 
selected sample of SDAs. Upon further consideration it 
was determined that two groups of SDAs would be 
surveyed: 1) a national sample of 12 SDAs, 
representing a structured mix of key SDA types; and 2) 
100 percent of the (12) SDAs in Washington State. 

The 12 SDAs in the national sample were selected 
through a process of consultation with national 
advisory committee members, state JTPA officials, 
National Alliance of Business researchers, and a 
network of other national, regional, and local 
contacts. The national sample was structured to 
include a mix of key SDA types in regard to the 
following variables: 1) geographic region of the U.S.; 

2) magnitude of JTPA funding (size of SDA II-A grant); 

3) population density (urban, suburban, rural); 4) 
jurisdictional configuration (city, county 
multi-county, etc.); and 5) apparent involvement to 
aate in local-level evaluation initiatives. 

A descriptive breakdown of the national sample by four 
primary criteria is provided below 

1) Size of II-A Grant (PY 1984) 

Over $6 million 2 SDAs 

$2-$6 million 3 SDAs 

Under $2 million 7 SDAs 

2) Population Density 

predominantly Urban/metro 4 SDAs 

predominantly Suburban 2 SDAs 

predominantly Rural 3 SDAs 

Mixed 3 SDAs 
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3) Jurisdictional Configuration 

City/County 3 gDAs 

County or Balance of County 2 SDAs 

Multi-County 5 SDAs 

Collection of Townships l SDA 

4) Sophistication: Indication of 
Substantial Involvement in 
Local Evaluation Initiatives 



Yes 
No. 



.6 SDAs 
.6 SDAs 



The National sample includes the following SDAs: 

1. Metro-Southwest SDA, Massachusetts )Region I) 

2. Cumberland County SDA, Maine (Region II) 

3. ^ Balance of Onondaga County SDA, New York (Region 

4. Baltimore County SDA, Maryland (Region III) 
(Region t IV) Fl ° rida Em P lQ y ment and Training Consortium 

S:.-? Ulf . C ? ast Business Services Corporation 
.uabissippi (Region IV) p auon ' 

7. Lansing Tri-County Consortium, Michigan (Region V) 

8. City of St. Paul, Minnesota (Region V) 

9. ^ Balance of Captital Planning Region, Texas (Region 

10. SDA V, Iowa (Region VII) 

(R; 8 ?on V Vi r n) Pl0yn,ent Trs " in 8 Consortium, Colorado 

Consort. & r „L e %%„ ^ 

1*1,11 """".ton SDAs, which were included in the 
survey, range from a large metropolitan system with a 

III' „"h nt a' IT-l^'Lrir^ 1 J""!, eight-count 
comparison with the 8 „atL„^ ^mpl^^ngton"^^ 
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were somewhat less likely to have conducted substantial 
local evaluation initiatives (only 4 of the 12 „ 

Th.;: ? substantial experience in Ms 

area.; There were also more multi-county SDAs in the 
Washington sample (8 of 12), fewer predominantly urban 
or metropolitan SDAs (3 of 12), and more SDAs with a 
predominantly rural or mixed urban/rural population (8 

The Washington SDAs surveyed include: 

SDA I. Olympic Consortium 

SDA II. Pacific Mountain Consortium 

SDA III. Northwest Washington 

SDA IV. Snohomish County 

SDA V. Seattle-King County 

SDA VI. Tacoma-Pierce County 

SDA VII. Southwest Washington Consortium 

SDA VIII. Pentad Consortium 

SDA IX. Tri-Valley Consortium 

SDA X. Eastern Job Training Partnership 

SDA XI. Benton, Frankin, Walla Walla Counties 

SDA XII. Spokane City and County Consortium 

It was the goal of this survey to tap input from a 
variety of local constituencies, including: SDA-level 
administrative staff; PIC members; and local elected 
cni ?i ?■ • paSt ex P erien ce with research at the 

bDA level it was anticipated that the greatest amount 
ot useable information would be derived from interviews 
with SDA administrative entity staff. It was 
therefore, determined that SDA directors and/or 

^r^f St3ff W ° Uld be the P™ry information 
source for the survey. 

The survey approach included initial contact with the 
administrative director in each SDA. This was followed 
by a scheduled telephone interview with the director or 

about P ISa°^ de5i 8 n ^ ed as being most knowledgeable 
SDA -level evaluation issues. In conducting SDA 
respondent interviews, we used structured interview 
guides and reporting formats, which included a 



combination of open-ended, limited choice, and scaled 
rating response items. SDA respondent interviews 
lasted between 4C-90 minutes, depending upon the time 
availability and interest of the respondent and the 
extensiveness of the evaluation issues and activities 
currently on the SDA agenda. 

We asked SDA directors to recommend PIC members and 
local elected officials with an interest or involvement 
in JTPA evaluation. All SDA respondents provided 
contact information for at least one PIC or local 
government representative. A shorter interview guide 
and reporting format, which duplicated some of the 
items on the SDA instrument, was used in conducting the 
PIC/LEO interviews. These interviews typically were 
accomplished within 30 minutes. The general level of 
awareness and ability to address specific evaluation 
issues was, understandably, lower among the PIC/LEO 
respondents . 

Interview reporting formats were completed for a total 
of 49 individuals, including: 12 Washington SDA 
respondents; 12 SDA respondents from the national 
sample; 15 PIC members; and 10 local elected officials. 

Recorded responses were analyzed, by subject area and 
on an item-by-item basis, for each of the major 
categories of respondents and across all respondents. 
Open-ended responses were analyzed and grouped by 
frequency; multiple-choice responses were tabulated; 
ana mean ratings were calculated for scaled ratine 
items. & 

In capturing, analyzing, and reporting telephone survey 
information, researcher accuracy, understanding, and 
interpretation are obviously open to question. While 
recognizing our own limitations, we take full 
responsibility for the material contained in this 
report . 
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OVERVIEW OF THE LOCAL CLIMATE FOR EVALUATION 



It is the goal of this project to produce evaluation 
design materials that will stimulate, guide, and 
support JTPA evaluation efforts - at the local SDA 
level, as well as at the state level. 

Receptivity for the products this project is developing 
cannot be taken for granted. Some of the factors that 
will determine whether our products are used at the SDA 
level are : 



1) The extent to which 
interested in evaluation 
evaluation initiatives; 



local constituencies are 
issues and willing to pursue 



2) The extent to which local systems are capable of 
supporting evaluation initiatives; 

3) The extent to which there are felt needs for outside 
assistance i n conceptualizing, designing and 
structuring local evaluation initiatives; 

4) The extent to which the products we develop fit - 
or can be fitted to - the particular priorities, needs 
and capabilities of local constituencies. 



This survey found considerable interest in evaluation 
issues among local constituencies. It also found 
considerable variation in how these issues have been 
conceptualized and acted upon to date at the SDA level. 

One trend was strong across the survey sample. Local 
accountability for performance is a key feature of the 
JTPA environment. Virtually all of the respondents 
reported that tracking and monitoring program 
performance is a high priority for their SDA. At a 
base level, all local constituencies expressed a 
concern for achieving and documenting job placements 
(and other positive outcomes) for participants at a 
reasonable cost, and all of the SDAs have some system 
for basic performance tracking. 

In most SDAs, concern for evaluation extends beyond 
performance tracking and documentation. The local 
constituencies we surveyed are interested in capturing 
feedback - both quantitative and qualitative - on 
program performance and effects; analyzing this 
feedback, extracting useful findings, and applying them 
in a variety of contexts. 



This analysis, and utilization of program information 
is conducted by SDAs at varying levels 8 of formally 
sophistication, and consistency. Such activity is often 
geared toward immediate funding or design decisions or 
ongoing management efforts which may draw on other 
types of information, such as labor market and 
demographic data, national R & D findings, as well as 

outcomes 86 ""^ feedback °" program operations and 
outcomes. SDAs are more likely to classify these 

?h£i V ii" 8 *? 3 " aly n iS ' as sessment, or even planning, 
than as evaluation . 5 

For many local-level actors, the term evaluation 
connotes a formal study or structured review of he 
system at large or of specific programs or components, 
conducted at a specific point in time, by persons 
Iffllit ?k- ongoing management and planning 

tUtl J:, 3 , type ° f eval ^tion, while viewed as 
worthwhile and potentially useful, has been less 
trequently employed among the SDAs surveyed. 

Definitional and conceptual confusion made it somewhat 
difficult to gauge quickly the extent to which each SDA 
has actually engaged in local evaluation initiatives to 
date. Upon initial inquiry, one-half of the national 
sample and nearly two-thirds of the Washington SDAs 
reported that they had not yet conducted substantial 
evaluations under JTPA. On further inquiry, it became 
apparent, however, that virtually all of the SDAs were 
capturing, analyzing, and using certain types of 

P !k f "cM Ce data 0n ' at least « an ad hoc basis. The 
other SDAs - half of the national sample and one-third 
of the Washington systems - were able to describe 
specific evaluation activities which had been locally 
designed and initiated. ' 

Much of the more developed evaluation activity we 
encountered was found among the larger SDAs in the 
sample When it comes to local capabilities for 
supporting evaluation initiatives, larger SDAs have 
some distinct advantages. These include: greater 
financial resources and more flexibility i„ 
administrative budgeting; more staff, including staff 
with needed technical expertise; a larger data basej 
and, m some cases, more sophisticated data retrieval 
ano analysis capabilities. However, we also 

STI' ,! 3 ;" mber ,° f Sn,aller SDA *' i^luding some 
tllr* ri & i locatlons of u "der $1 million, which had 

t.. 6 1Ve , W3yS t0 C3rry out Really tailored 
evaluation initiatives. 

Even with current limitations on resources and less 
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than optimal capabilities, the majority of SDA systems 
have the basic capabilities, interest, and desire to do 
more in the area of evaluation. Over 80 percent of the 
SDAs surveyed have automated data storage and retrieval 
systems which are capable of, or can be adapted to 
support increased evaluation efforts. Over 70 percent 
have some kind of system in place for capturing 
follow-up information on post-program (13 weeks or 
longer) outcomes. and 75 percent indicated specific 
plans to upgrade or expand their evaluation efforts 
during the coming year. 

Major areas of development in local-level evaluation 
include : 

1) Instituting longer-term follow-up and/or expanding 
or enriching follow-up contacts with employers or 
participants; 

2) Conducting more systematic and detailed analyses and 
making greater use of program and follow-up data; 

3) Upgrading the MIS or implementing a new information 
system; 

4) Conducting process evaluations or special assessment 
studies ; 

5, x Creating new linkages for accessing and using 
non-JTPA data bases ( welf are/U . I . ) 

Finally, our survey found that local constituencies do 
recognize a need for outside assistance and guidance on 
evaluation designs and techniques. This need was 
expressed particularly by smaller systems which lack 
the expertise or resources to develop tailor made 
approaches from the ground-up. Nearly 30 percent o2 
the SDAs surveyed had solicited some outside assistance 
m designing or conducting evaluations. Even more 
sophisticated, larger systems which are confident of 
their own technical expertise expressed a desire for 
high quality "off-the-shelf" designs and technical 
assistance guides which could be adapted to local 
purposes . 

This does not mean, however that local constituents 
are likely to jump into implementing any odel that 
might be offered. The bottoraline criteria for 
acceptable designs are simplicity, practicality, and 
reasonable prospects of producing outputs tailored to 
local-level uses. SDAs already feel burdened by 
state-imposed reporting and follow-up systems which are 
not geared to readily produce the types of information 



that would be most useful for local purposes. 

Local-level needs for evaluation information and 
analyses are differed f rora national or "ate-level 
needs, because local constituencies are direc^lv 
involved in managing and awarding contracts developing 
policy frameworks, allocaLing resources, ana designing 
and delivering services. aesignmg 

The next section of this report looks at lccal 
priorities and needs i„ reg ard o specific uses of 
evaluation information. s of 
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LOCAL USE OF EVALUATION INFORMATION 



1) Per formance Management and Corrective Action 
Survey responses indicate that the most common use to 
date of program performance data generated at the 
local level has been in the ongoing management of 
program operations and contracts, the identification of 
problems, and the initiation of corrective action 
efforts. Identifying performance problems and taking 
prompt action to correct them was cited as the most 
important current use of evaluation information by SDA 
directors and staff. (PIC members and LEO respondents 
tended to rate this use as slightly less important.) 

The thrust of ongoing management efforts at the SDA 
level tends to be double-pronged: 1) to ensure that 
actual levels of participant and financial transactions 
are synchronized with levels specified in the job 
training plan and individual contracts; and 2) to 
ensure that the system as a whole will meet or exceed 
its annual performance standards. 

Most SDAs prepare monthly managerial reports which 
analyze, at minimum, actual expenditures, enrollments 
participant characteristics, and termination outcomes 
in comparison with planned levels - for the program 
overall and for each major training component and/or 
contractor. Performance data is summarized in 
quarterly reports which measure system-wide performance 
with regard to each of the basic performance standard 
indicators. 

In the review of program performance data for ongoing 
systems management, particular attention is paid to the 
numier and rate of tcrminees entering employment. 
Placement performance is fairly easy to track and 
analyze at any point in time on a contractor or 
component basis. Managerial action is taken promptly 
when placements are lagging seriously behind planned 
levels. 

Controlling costs is another key management concern. 
Computing and tracking cost per entered employment (or 
cost per positive terminacion for youth) on an ongoing 
basis presents some difficulties for most SDAs 

?5?c? C i al d3ta is not integrated with participant-based 
(.Mlb) data in the lar g e majority of SDA systems. Cost 
data sometimes lags far behind information on 
participant transactions. Some SDAs only compute unit 
cost measures o* a quarterly or year-end basis. Unit 
costs are often controlled through ceilings written 
into cost reimbursable contracts or (more effectively) 
through the provisions of unit price contracts, whi-h 



pay a fixed amount per placement. Larger SDAs with 
fixed unit price contracting systems generally maintain 
separate cost accounting/ verification systems to track 
on a weekly basis contractor payments linked with 
specific participant transactions. It is the 
exceptional SDA (only 2 of the 24 surveyed) that has a 
fully integrated financial and participant-based 
information system. 

Computing actual versus planned performance statistics 
for basic indicators is less difficult for many SDAs 
than analyzing on an ongoing basis how actual 
performance stacks up against state-applied performance 
standards. State use of the regression model in 
adjusting performance standards for each SDA means that 
performance standards may be considerably altered by 
unanticipated changes in the program. For example, a 
shift in the demographic composition of enrollees or in 
the average duration of enrollment will alter the cost 
per entered employment standard against which the SDA 
will be evaluated by the state. In order to calculate 
how actual performance compares with the performance 
standard at a given point in time, an SDA must be able 
to accurately compute both actual unit costs and the 
adjusted performance standard. Many SDAs lack this 
capability at the current time. They concentrate, 
instead, on managing performance to correspond as 
closely as possible with the plan and wait until the 
end of the year to see how they will "come out" in 
regard to the cost per entered employment and cost per 
positive youth termination performance standards. 

Another important focus of program management is 
monitoring performance in regard to key target groups. 
In some SDAs, concern is limited to groups for which 
specific enrollment levels have been mandated by the 
state (such as dropouts or WIN registrants.) But at 
least half of the SDAs surveyed have placed a high 
priority on achieving at least specific enrollment 
leveis for other "most in need" target groups (such as 
minorities, single parents, and persons with 
handicaps.) With the exception of welfare recipients 
(for whom the entered employment rate is a mandated 
performance standard), management analysis of target 
group performance is typically limited to enrollment, 
as opposed to termination, analysis. About half of the 
SDAs reported that correct! ve action procedures are 
initiated when program contractors fail to meet target 
group enrollment objectives. 

Other key performance indicators in program management 
are the average wage at placement and the percentage of 
placements that are training related. At least 



one-third of the SDAs surveyed have set specific 
objectives in one or both of these areas for major 
program components or individual training projects. 
These objectives are based in part on a detailed 
analysis of actual placement data. 

Job retention is an increasing concern in performance 
management at the SDA-level. Most of the SDA systems 
included in the survey track 30 day retention data, and 
at least one-quarter of the SDAs pay explicit attention 
to this indicator in ongoing program management. Only 
a very few SDAs, however, incorporate longer-term 
follow-up data in managerial performance reviews. 
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2) Input for Local Policy Development 

Survey responses indicate that the usefulness of 
evaluation information in orienting, educating, and 
informing local policymakers has been second in 
importance only to its usefulness in ongoing systems 
management. As a group, PIC members tended to consider 
this use of evaluation information to be the most 
important use to date at the SDA-level. In most SDAs, 
the PIC bears the primary responsibility for 
establishing local policy goals and program objectives. 

The establishment of a meaningful SDA policy framework 
requires a good understanding of the local program 
environment. At a minimum , PIC policymakers have been 
required to understand the national performance 
standard indicators and determine the extent to which 
JTPA programs in the SDA will be required to meet 
specific performance goals. An analysis of first year 
performance results has provided baseliiie information 
for setting specific program objectives for subsequent 
years in most of the SDAs surveyed. 

In some SDAs - perhaps half of those surveyed - local 
policy development is limited to the goal of exceeding 
all state-levied performance standards, thereby 
demonstrating superior performance in comparison with 
other SDAs in the state. 

While SDAs with more developed policy frameworks also 
take national/state performance standards into account, 
their policy goals tend to reflect local circumstances 
and preferences. Several of the SDAs surveyed have made 
explicit policy choices to place greater priority on 
providing high quality training in specific 
occupational areas or on serving specified "most in 
need" segments of the eligible population than on 
achieving the cost per entered employment standard 
specified by the state. Some PICs are setting 



stringent objectives for target group enrollment, 
training-related placement rates, retention rates 
average wage at placement, or ceilings on the 
proportion of direct placements v/hich will be allowed 
These objectives are designed to reinforce local policy 
orientations. 

The base of information and understanding required to 
support the development of a local policy framework 
goes beyond exposure to statistical summaries of 
enrollments, costs, and termination outcomes. Analysis 
of information not specifically generated through 
program activity (such as demographic and labor market 
data or national R&D findings) clearly plays a role in 
framing a local policy orientation. But perhaps more 
important is a substantive assessment of local program 
design and effectiveness with regard to key target 
populations and targeted occupations or sectors of the 
labor market. PIC members and SDA staff report that 
process evaluations, case studies, and other assessment 
efforts that provide descriptive and qualitative 
feedback as well as quantitative analysis have been 
extremely valuable in giving policymakers a clear 
picture of who the programs are serving, what they are 
actually doing, and what outcomes and impacts they can 
be expected to achieve. 



3) Program Funding and Design 

Local constituencies make decisions on how SDA 
resources will be allocated or deployed and on what 
programs, services, or contractors will be funded. In 
some cases SDA-level roles in program design and 
development are limited to setting performance goals, 
allocating funds, selecting contractors, and 
negotiating contracts. But in most cases PICs and SDA 
staff have a more extensive role in shaping program 
design. As shown by a recent survey conducted by the 
National Alliance of Business, administrative entities 
are directly involved in program operations in over 
half of all SDAs. 

It is in the areas of program funding, design, and 
development that evaluation information is viewed by 
local constituencies a* having its greatest potential 
usefulness . 

A number of the SDAs we surveyed ar~ using locally 
generated performance lata as a basis for allocating 
training dollars across functional service areas. 
Several are using linear projection models, which use 
past performance data as a basis foi predicting the 



performance outcomes (enrollments, placements, unit 
costs, etc.) which can be expected from different 
dollar allocations to functional service categories 
(OJT, institutional skills training, job club/placement 
services, etc.) The SDA selects and uses the 
allocation formula most likely to maximize performance 
results. While more sophisticated allocation models 
may include factors such as service to key target 
populations, wage rates, and training-related outcomes 
in their performance projections, none to date have 
included longer-term employment or earning gains for 
participants or reductions in welfare payments. 

One of the simplest and most prevalent uses of 
evaluation information has been in making annual 
funding decisions with regard to specific contractors 
or projects. PIC committees analyze and compare past 
performance results in considering contract renewals 
and funding levels. Given the grant reductions being 
experienced by most SDAs, funding decisions often 
revolve around how necessary cutbacks will be 
distributed across current contractors or projects. 
Those training projects or program operators who do not 
compare favorably with their competitors in regard to 
job placement/positive termination or unit cost 
performance are the most likely to have their funding 
cut back or eliminated. Here again, while factors such 
as ability to serve high priority target populations or 
to provide priority services not otherwise available 
ar - generally considered in such funding decisions, 
3 formation on probable longer-term impact and 
effectiveness is typically not available to 
decision-makers . 

SDA staff and PICs become more substantially involved 
in program design through such activities' as: the 
development of RFP criteria, programmatic 

specifications, contract provisions, and youth 
employment competency standards; the selection of 
occupational areas for institutional skills training; 
the development of training projects and curriculum. 

In thase endeavors, local constituencies are most apt 
to be concerned with determining the types and levels 
of skills, knowledge, and behavioral traits associated 
with securing and maintaining employment (often in & 
specific sector of the job market) and with identifying 
the best training approaches and techniques for 
assisting various types of JTPA-eligibles to secure 
needed competencies and make a successful labor market 
adjustment. SDAs are also interested in identifying 
those industries, occupations, or types of employers in 
the local market which offer the best prospects for 



very limited segment of the community. There does not 
appear to be much community interest in JTPA in 
general, and there is even less interest or 
understanding for the types of findings typically 
generated by local assessment efforts. Descriptive 
reports of successful economic development linkages or 
special projects with a strong human interest factor 
are more apt to generate media attention. 



5) Marketing 

Survey respondents - particularly PIC members as a 
group and SDA administrators from Washington State - 
placed a relatively high priority on demonstrating the 
utility and benefits of JTPA programs to local 
employers. But few SDAs to date had managed to 
effectively develop or package evaluation information 
for use m marketing the program to employers. 

PICs and SDA administrative entities have an expressed 
interest in generating broader private sector support 
f °£ employment and training efforts targeted on 
JTPA-eligibles. But SDA-level marketing campaigns and 
materials have tended to use only generalized messages 
t£t>a rtlr ' 8 em P lo y ers about the potential benefits of 
JTPA involvement. This is particularly true for SDAs 
which contract out all service delivery functions and 
have little or no contact with the employers who 
actually train and employ JTPA participants. 

However, a significant portion of SDA administrative 
entities (over 40 percent according to the NAB survey) 
are directly involved in the delivery of JTPA placement 
services. Our survey responses indicate that SDAs with 
this kind of service delivery role are more likely to 
carefully analyze placement data and use their findings 
as a basis for targeting training, job development, and 
placement efforts. 

Survey respondents also indicate a growing trend toward 
SDA-initiated contacts with, employers as a means of 
generating useful feedback on employer satisfaction and 
program effectiveness. Such contact mechanisms could 
be used to generate evaluation information tailored for 
future marketing efforts. 



PRIORITIES FOR OUTCOMES MEASUREMENT AND ANALYSIS 



Survey responses show that in analyzing evaluative 
information for program planning, management, and 
funding, SDAs to date have relied heavily on a fairly 
limited array of short-term indicators. However, the 
survey found a growing interest at the SDA level in 
measuring and analyzing longer-term outcomes of various 
types. A number of the SDAs are already investing 
considerable resources in capturing follow-up data. 

We asked SDA respondents to rate the level of 
importance for their SDA/PIC of information on a 
variety of possible outcomes measures. Respondents 
rated each measure proposed on a scale of 1-5 (with 
l=of no importance and 5=extremely important.) We 
compiled mean ratings on each measure for both the 
national sample, the Washington SDAs, and for the 
combined sample. Priority measures are listed below 
(with comments) in order of their composite mean 
rating. 



1} ^ Stability - Retention of Employment with the 

Placement Employer f Mean Rating = A i ) ' 

This outcome measure received the highest overall mean 
rating, but it was rated somewhat higher by Washington 
SDA respondents than by those in the national sample. 
Most SDAs view job retention as an interim rather than 
a final indicator of program outcomes. (Terminees are 
not expected to remain employees of the placement 
employers on a lifetime basis.) But retention data at 
13 weeks is becoming more readily available and is 
viewed as a highly useful indicator of training quality 
and program effectiveness 



2 ) Differentials in Results/Outcomes by Servi ce 
Strategy or Project fMean Rating = 3.9) 

SDA respondents from the national sample were more 
likely than Washington respondents to place a high 
degree of importance on ability at the SDA level to 
measure differentials in outcomes or results across 
service strategies and projects. Most likely this is 
because of the greater number of smaller systems within 
Washington State which fund fewer individual projects, 
serve fewer participants, and tend to make less 
distinction among service strategies in program funding 
and assessment. A number of SDA respondents stressed 



the opinion that measures of differentials in outcomes 
or results by service strategy /pro ject must take into 
account differentials in the types and characteristics 
of participants served. 

3 ) Particip ant Earning Gains or Losses f3.al 

Few of the SDAs surveyed are currently able to estimate 
the extent to which participants experience an increase 
m annual earnings after program termination (as 
compared to pre-program experience.) None of the SDA 
respondents was interested in conducting net-impact 
evaluation at the local level, but most would like to 
have better information on participants' post-program 
earnings. In wage reporting states, U.I. data was 
viewed as the best potential source of post-program 
earnings data. Follow-up contact with participants was 
not generally viewed as a very accurate source of 
information on annual earnings. Annual earnings gains 
were viewed as a more meaningful measure than 
comparisons of pre/post hourly wages. But a number of 
respondents were interested in the extent to which 
participnat wage levels change during the 
post-placement year. 

$} liaining Relatedness of Employ ment 

Outcomes ( ^ y ^ c ' 

This measure received a higher mean ranking from the 
national sample SDAs than it did from Washington 
respondents. The extent to which post-program jobs are 
related to the type of training provided is of greater 
concern in SDAs which devote a substantial portion of 
their JTPA dollars to occupational skills training 
projects. These SDAs are interested in whether 
participants remain employed in training-related 
occupations and how they fare in targeted occupational 
areas after initial placement. 

5 ) Reductio ns in Welfare Payments O. 5^ 

Only two of the SDAs surveyed reported that they were 
currently estimating or me^uring reductions in welfare 
payments. Most were only calculating welfare entered 
employment rates. The ability to accuiavely calculate 
reductions m welfare payments depends upon access to 
welfare data, and many respondent had become 
discouraged in their attempts to gain such access. 
Some expressed a doubt that such access would ever be 
achieved at the local level and faulted th . state for 
tailure to secure cooperation from the welfare 
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PRIORITIES FOR EVALUATION DESIGN PACKAGES 



In an attempt to focus input from local constituencies 
more specifically on potential evaluational models we 
asked survey respondents to rate the level of 
usefulness or importance to their SDA of various 
elements that might be included in a total evaluation 

Their ratings (on a 1-5 scale) for each potential 
element were averaged, overall and for each of the 
following groups: 1) national sample SDA respondents; 
2) Washington State SDA respondents; 3) PIC members; 
and 4) local elected officials. Responses are 
summarized below, in the order of priority indicated by 
overall mean ratings. 



1) Strategies for evaluating program effectivene ss in 

achieving longer-ter m employment and earnings g ai"nV7^7 

participants (Me a n Rating = 4.4) ' 

The need for evaluation designs which offer a 
practicable approach for SDA-level evaluation of 
program effectiveness in achieving longer-term 
employment and earnings gains for participants was 
raued most important by both national and Washington 
SDA respondents. PIC members gave this element a mean 
rating of 5 (on a 1-5 scale.) (The mean rating ojven 
by local elected officials was only 3.7.) 

The interest of SDAs in gaining insights on longer-term 
program effectiveness is evidenced by the number of 
survey sites (almost half) which reported involvement 
in new follow-up sctivities and plans. But the 
majority of SDAs seem uncertain how best to structure 
such efforts and use the information they yield. 

c Strategies for Identifying Causes of Poo r 

Perf ormance (4.3) 

All local constituencies tend to place some premium on 
evaluation approaches which offer the potential for 
identifying and predicting factors associated with poor 
performance. Only local elected officials, howeSer, 
I' h ". evaluation element a top mean rating (5). 
inrf^- systems are typically geared to pick up 
indications of possible poor performance results fairly 
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early in the program year. But administrative staff 
often have a difficult time determining the actual 
causes or conditions responsible for poor performance 
showings and ameliorating these conditions. Staff have 
even less ability to predict in advance which 
"S!" 1 ;? 11 !" 3 . or t programmatic variables are most 
crucially linked with perfcrmance results. 

^ l ^ r ftep,ies for evaluat ing program effec tivene ss and 
peneti ts from an employer perspective (4.2~ V 

This evaluation element tended to be rated somewhat 
higher in importance by local elected officials and PIC 
members than by SDA respondents. SDA respondents do 
place a relatively high priority on evaluation 
information that could be used for employer marketing. 
But some respondents believe that attempts to evaluate 
program impact on employing firms would be impractical 
and would prove less useful than qualitative feedback 
on employer satisfaction. SDAs could use assistance in 
techniques for tapping and analyzing employer feedback 
ana using this information in marketing. 

* } Techniques for analy zing rh» relationship hP^ .n 
program strategies and performance results f4) 

No SDA can afford to ignore performance standards and 
no SDA director wants to be stuck with planned 
performance objectives which can't possibly be aet 
through the mix of program strategies or services that 
have been funded SDA respondents place a high premium 
on evaluation designs which offer mechanisms (such as 
linear projection models) for predicting the affect 
ITJ 0 Ser \ lce raix ' targeting strategy, and other program 
" decisions are likely to have on performance 

il?!lllhl. I n °/ a PP roaches » while not seen as 

infallible, are viewed as giving administrative 
entities greater control over performance outcomes. 

II Practicable approaches for evalua ting the hPn.fUc 
of JTPA f or the community f3 T jn ——a 

All of the groups surveyed tended to rate the ability 
to evaluate and demonstrate the benefits of JTPA for 
the community as being an, at least, moderately 
i!Sn r i«?- eleraent u , in * n overall SDA evaluation design.: 
for vjr ngS f °K thiS fcval * ation element were highLt 
respondents l0 ^ St f ° T ""Kington SDA 



No one we surveyed felt that a net impact evalna^on 
model was practicable at the SDA level. (0-e SDK 
however, was establishing the kind of linkage^ to 
automated U.I. and welfare data bases which would make 
some form of net impact assessment feasible.) But 

TTD?? nde ^ S did f6el *■* was important to demonstrate 
J1PA s effectiveness as an investment strategy and its 
beneficial effects for the community at large. 

6 > Approaches that allow for assessment of the 
effectiveness JF_ a speci f ic program s trategy oT 
c omponent (3.8) 

Respondents felt that an overall evaluation design 
packar ^ilored to SDA-level use should include 
appro: for the intensive assessment of a specific 

stratr?), omponent, or project. SDAs tend to focus 
evalua. u ,d geared toward program upgrading or redesign 
on only specific elements of the system; rarely is the 
total system up for grabs at any given moment in time. 
Process evaluation models which combine qualitative and 
quantitative techniques are viewed as being appropriate 
for intensive single-component assessments at 'the SDA 
level . 



7 ) Techniques for comparing program strategies and 
results across contractors (3.4) 

In a climate that places a high premium on performance 
but offers a declining base of resources for funding 
program services, valid techniques for rating 
contractor effectiveness are increasingly important at 
the SDA level. Larger systems with a number of 
competitive contractors tend to operate in a political 
environment which focuses considerable scrutiny on 
funding decisions. Most respondents realized that 
comparison of performance across contractors should 
take into account the characteristics of those b^ing 
served and the types of services being provided. 



8) Techniques fo r evaluating local processes - such a s 

planning, managing and contracting approaches - T77 

assessin g, the affects of these processes on DroTraTT 
design and quality (3.31 ' " R 

ong the groups surveyed, local elected officials and 
Washington SDA respondents were more apt to place a 
relatively high degree of importance on this element of 
local-level evaluation. Several SDAs in the national 
sample reported that they had conducted limited 
evaluations of specific local processes which had 
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produced useful information. But other SDA respondents 

felt that self-evaluation of this type was apt to be 

less than objective, and most preferred to focus their 

limited evaluation capabilities on training strate~ies 
and program outcomes. 

9) Approaches for evaluating the effectiveness of 
coordination li nkages with non-JTPA programs (2.9) 

Among all the groups surveyed, only local elected 
officials and PIC members rated the assessment of 
coordination linkages as a relatively important element 
m an SDA-level evaluation design package. Most SDA 
respondents had a hard time envisioning practical 
evaluation approaches which might yield useful insights 
on the effectiveness of coordination efforts. And a 
number of respondents indicated that meaningful 
coordination with non-JTPA programs was more dependent 
upon action taken at the state level rather than the 
local le il. 

10) Ability .o analyze SDA effectiveness in compa rison 
with other SDA systems : 

We did not explicitly ask for a rating on this 
potential feature of an overall evaluation design 
package. But many respondents mentioned that they 
would welcome the adoption of models which allow some 
ability to compare their effectiveness in various areas 
with that of other SDAs. Several respondents mentioned 
that the validity of such comparisons would depend upon 
a uniform base of definitions for key reporting 
categories, such as "enrollment" and "placement". 
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SUGGESTIONS FOR EVALUATION MODEL DESIGNERS 



We asked survey respondents to tell us (on an 
open-ended basis) what they felt were the most 
important considerations evaluation designers should 
bear in mind in developing models or guides which would 
be useful at the SDA level. Their responses were 
recorded, grouped, and analyzed. A summary of 
suggestions from local constituencies is offered below. 



1 ) Focus on specific purposes or *j a & 

Local constituencies are not likely to adopt evaluation 
models proposed by outside designers without a clear 
sense of how the outputs of suggested evaluation 
designs can be used at the local level. A number of 
respondents suggested that evaluation guides begin with 
specific purposes or local utilization uses (such as 
better targeting of program services) ; indicate the 
kinds of evaluation information needed to effectively 
address these purposes; and then go on to suggest 
appropriate methodologies for securing and analyzing 
evaluation information. 

Evaluation models which are capable of generating 
findings useful for a variety of different purposes 
will clearly be preferred. Specifically , local 
constituencies will want to know how proposed 
evaluation models will be of potential assistance to 
them in the following areas: 1 ) managing system 
performance; 2) developing locpl policy goals; 3) 
allocating resources; 4) designing, developing , and 
directing programs; 5)' publically documenting system 
accomplishments; and 6) marketing the program to local 
employers. 



2 ) Make g uides as practical, simple, 
possible 



and clear as 



In preparing written g 
clear explanation of 
uses, and limitations, 
high level of tech 
audience. Materials 

considerations and be written in a straight forward and 
noncondescending style. Even SDA representatives with 
a relatively hi C h degree of sophistication in 
evaluation methodologies will be put off by guides 
which appear to be highly academic and too far removed 



"ides, designers should 
the models, their key 
Designers should not 

'ical expertise across 
should emphasize 



offer a 
elements , 
assume a 

the SDA 
practical 
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from the program environment . The guides should be 
understandable to administrators and policymakers and 
useful to staff with various functions in the SDA 
system - planners, managers, fiscal/MIS staff - as well 
as evaluation specialists. The guides should clearly 
indicate those areas of design and implementation where 
special technical expertise or outside guidance is 
advisable . 



3) Encompass a ranae of design options 

SDAs vary widely in regard to local program focus, 
organizational and service delivery structures, and 
system capabilities. If the models are to be widely 
used at local level, they must include options for 
tailoring evaluation designs to local circumstances, 
capabilities, and needs. Specifically, respondents 
expressed major concern that some options be provided 
that are realistic for SDAs with small grants and very 
limited staff capabilities. 

While local constituencies expressed some concern for 
validity and reliability of results, they are more 
interested in capturing rich and useable feedback than 
in conducting rigorously scientific evaluations. SDA 
constituencies are wary of models which require 
experimental controls or complex sampling designs. 
They would like more practical options for collecting 
and analyzing feedback on program effectiveness and 
longer- terra outcomes. They would like the design 
flexibility to combine both qualitative end 
quantitative approaches. They would also like design 
packages which offer the flexibility to focus 
intensively on single program components or projects or 
to conduct broader systerawide assessments; to conduct a 
quick limited-purpose assessment or to incrementally 
implement a more extensive ongoing evaluation lystera. 



Consider bot h national performance standards and 
local goals 

SDA constituencies are concerned with analyzing and 
tracking system performance in regard to the national 
performance standards applied by the state in awarding 
performance incentive funds. They are also interested 
in evaluating the extent to which programs are 
achieving local policy goals. Models for ongoing 
evaluation initiatives at the local level should 
encompass the "uniform 11 performance standard indicators 
(including regression model adjustments, but they 
should also be flexible enough to allow SDAs to focus 
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specifically on those goals of particular concern to 
local policymakers . 



5) Back up models with technical assistance 

Respondents recognize that there is a limit to the 
extent to which national guides can offer technical 
advice geared to local-level adoption and 
implementation of the models. Many respondents felt 
that the prospects of local SDAs actually using the 
suggested models would be greatly enhanced by a focused 
technical assistance and training effort. 

SDA-level actors view evaluation initiatives with both 
interest and trepidation. Less sophisticated staff are 
particularly fearful of the potential technical 
difficulties involved in evaluation. Most SDA 
constituencies will need to be convinced that 
implementation of proposed evaluation approaches is 
both technically feasible and likely to produce outputs 
which are worth their efforts. Several respondents 
suggested regional workshops or on-site training 
provided by persons with actual experience in 
conducting successful evaluation initiatives at the 
local level . 



183 



CD ? r PAGE 29 

ML 



LOCAL SYSTEM CAPABILITIES AND CONTINGENCIES 



1 ) Data Col lection. Retrieval, and Analysis 

All of the SDAs surveyed have a management information 
system (MIS) which tracks program transactions 
(enrollments, training assignments, completions, 
terminations, and placement outcomes) and a variety of 
client characteristics and pre-prograra status measures 
on an individual participant basis. Most SDAs can 
break our. participant-based performance indicators by 
major component or contractor on a monthly basis. 

All but two of the SDAs operate separate systems for 
tracking financial transactions. While these systems 
allow for analysis of expenditures by major component 
or contractor, it is often difficult for SDAs to 
accurately compute indicators which integrate 
participant and financial data (such as cost per 
participant and cost per entered employment) on a 
current basis. Lack of integration in regard to 
financial and MIS data is viewed as a major limitation 
in local evaluation efforts by over one-third of the 
SDAs surveyed. 

The majority of the SDAs (all of the Washington SDAs 
and over 60 percent of the national sample) are using 
management information systems which have been, or are 
in the process being, adopted on a statewide basis. 
All but one SDA will be tied into a statewide MIS by 
the end of the next program year. About one-quarter of 
the national sample SDAs were using the Washington 
State datf, flex software. 

The I-rdware used for SDA MIS systems included: 
mainframe computers (used by a regional network of SDAs 
/o ^y t N° f 3 raunici P al system) (2 SDAs); mini computers 
(2 SDAs) ; micro computers (15 SDAs); and "dumb" 
terminals : i _d into a statewide computer (2 SDAs). 
Only one SDA was currently using a manual (card sort) 
A J. S • 



The SDAs most likely to be fully satisfied with their 
= .IS capabilities were those using systems developed 
especially for local level use, which had a good deal 
of xlexibiiity in data base management, retrieval and 
analysis. (These generally required mini or mainframe 
capabilities . ) 

SDAs using state-developed MIS packages with micro 
computers or dumb terminals were apt to complain that 
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these systems were geared toward meeting state-level 
reporting requirements, xather than meeting local 
information and analysis needs. But those SDAs which 
had enhanced their systems with added program 
capabilities were, by and large, satisfied with the 
statewide system. 

Only three SDAs reported that they were currently 
experiencing no problems with their MIS. The most 
commonly reported problems were: 1) lack of ability to 
break out needed data at a sufficient level of detail; 
2) need for manual transcription/computation of plan vs 
performance indicators; 3) not being programmed for 
performance standard calculations; 4) lack of staff 
expertise in HIS operation and programming ; and 5) 
system bugs and breakdowns. The majority of the SDAS 
were adjusting to new MIS systems. Many said they were 
still experimenting and had not yet fully tapped their 
systems T capabilities. 

When asked how well equipped their MIS was to support 
evaluation activities currently underway or planned for 
the future, one-eighth of the SDA respondents rated 
their MIS capabilities as excellent. Three-fourths of 
the SDAs reported that their basic systems were at 
least adequate - though about half felt that 
adjustments and additional programming expertise would 
be required, Only four SDAs rated their systems as 
poor or inadequate for supporting local evaluation 
initiatives (and two of these SDAs expected to change 
their system during the coming program year.) 

Most of the SDAs surveyed collect 30 day retention data 
on placement© f which is integrated in the MIS data 
base. About one-quarter of the SDAs currently collect 
no additional follcw-up data. About two-third collect 
or receive follow-up data on (at least a sample of) 
participants at 12 to 13 weeks following placement or 
termination. Only two of SDAs were currently 
collecting longer-terra (4-12 months) follow-up data. 

Most of the current follow-up systems have been 
initiated to meet state requirements, rather than to 
serve local purposes, according to SDA respondents. 
Only one-quarter of the SDAs are currently conducting 
locally designed follow-up efforts. Locally designed 
designed and initiated follow-ups are more likely to 
include employer-based as well as participant based 
contacts and to solicit qualitative feedback as well as 
objective data on employment retention and earnings. 
Only one SDA has a follow-up system which collects U.I. 
wage reporting data and welfare payments data both pre 
and post program termination. 
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Cloll tl h % r,1 ; tlV ! ly n6W eff ° rt for most SDAs. 
Close to half of those surveyed indicated that they 
planned either to initiate new local follow-up designs 
Iff n ll & * ?and ' u au Sment or improve existing follow-up 
efforts during the next six months. Only two of the 
•SDAs were currently integrating follow-up data in the 
MIS data base, but of number of others said they 
planned to do so. y 

A number of SDAs reported as major evaluation 
constraints their inability to access information from 
the Employment Service (on participant earnings or 
employment status) or the welfare department (on 
welfare payments to participants.) Few SDAs were 
attempting to measure pre/post program increases in 
earning or employment stability or reductions in 

rrpf ar %*? ayra ? ntS ' M ° St res P°ndents felt that state 
* officials should do more to expedite the release 
of such information. 

In the "data block" environment, the Denver information 
sharing system constitutes a notable exception. 
Colorado is a wage reporting state, and the Denver SDA, 
which already has access to some U.I. and welfare data 
will soon have the capability to directly interface 
with U.I. and welfare data bases, access needed pre and 
post program information on JTPA participants, and 
store this information in its own fully automated data 

D 3 S 6 • 



2) Funding Evaluation Initiai- 



lves 



Scarcity of funds for conducting evaluation was viewed 
as a major obstacle by over two-thirds of the SDA 
respondents. SDAS with smaller II-A grants (under $2 
million) found it especially difficult to set aside 
funds for conducting program evaluation activities when 
administrative budgets were already stretched to the 
limit. Even SDAs with larger II-A grants, greater 
administrative budgeting flexibility , and a history of 
substantial evaluation activity, reported considerable 
difficulty in breaking out funds for new evaluation 
mitiati/es. 

Evaluation activities are often tied into o-her 
functions of the administrative entity, and many SDA 
respondents have some difficulty estimating dollar 
expenditures for specific evaluation initiates. 
There is clearly a wide range of variation ir> the 
amounts that individual SDAs have expended on 
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evaluation to date. Larger SDAs reported expenditures 
from annual administrative budgets of $15,000 to 
$50,000 to support ongoing follow-up efforts. A 
smaller SDA reported spending $6,000 on an assessment 
of current program effectiveness in serving 
harder-to-serve target groups. Another SDA of about 
the same size spent only 80 man hours (about $1,000) on 
an assessment of an institutional skills training 
project . 

Some SDAs have found creative ways to fund follow-up 
efforts and other evaluation initiatives. One SDA 
director was successful in enlisting the cooperation of 
business school staff from a local university in 
developing a design for program performance review and 
follow-up and using graduate students to conduct the 
study. Another SDA with a Title II-A grant of less 
than $1 million got assistance from the National 
Alliance of Business in designing and conducting a 
program effectiveness assessment, with $3,000 in SDA 
resources matched by a $3,000 grant from the state. 

Resource limitations are a major consideration in the 
design and implementation of evaluation activities, but 
it is clear that scarcity of funds need not preclude 
SDA-level evaluation. 



3) Staffing Evaluation Initiatives 

Capabilities for designing and staffing local 
evaluation initiatives vary widely among the SDAs 
surveyed. While one of the largest SDAs has over 60 
persons on staff (some of which are involved in service 
delivery,) several of the smaller SDAs have only 2-3 
staff persons in the entire administrative entity. 

The bulk of SDA-level staff are deployed in contracts 
management, financial accounting, MIS, and PIC 
support. Only a handful (3 of the 23) of the SDAs we 
surveyed have one or more staff persons designated as 
evaluation specialists, and these are all larger 
systems. 

While relatively few (less than 20 percent) of the SDAs 
surveyed admitted to s serious lack of local expertise 
for conducting evaluation, it is clear that much of the 
local evaluation activity is carried out by generalists 
or others with limited technical background in 
evaluation . 

In about one-third of the SDA systems, other staff - 
specialists, planners, program monitors, 
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administrators - are charged with responsibility for 
evaluation-related activities. A number of respondents 
felt that there are clear advantages to allocating 
evaluation roles across various staff units, rather 
than vesting evaluation functions in an independent 
unit removed from ongoing system management and 
planning. But in over half of the SDAs surveyed no 
one on the staff is charged with ongoing responsibility 
for program evaluation. 



At least half of the SDAs that reported conducting 
follow-up contacts, intensive program reviews, or 
r cial assessment studies, have relied heavily on 
ouLSide assistance. Outside resources used by SDA 
respondents included paid consultants, public interest 
organizations, student interns, university staff and 
graduate students, and, in several cases, proeram 
operators. & 



*) PIC Roles & Support for Evaluation 

In the majority (two-thirds) of the SDAs surveyed, the 
Fnvate Industry Council has a program evaluation or 
oversight committee which meets on at least a quarterly 
basis. In all of the SDAs, information on system 
performance is presented to the PIC on at least a 
quarterly basis. The emphasis to date in PIC oversight 
roles has been on performance monitoring, rather than 
effectiveness evaluation. 

In most cases PIC members review plan vs performance 
statistics prepared by SDA staff, respond to 
performance problems redflagged by staff, and initiate 
or approve corrective action strategies. PIC Members 
are likely to become more substantivly involved in 
assessment issues through participation on program 
planning or funding committees, or subcommittees that 
focus on specific program components or issues. 

PIC members would like better information for policy 
development, program funding, and design decisions, but 
they lack any clear picture of just how local 
evaluation efforts could improve the base of 
information and understanding. They tend to equate 
evaluation with statistical reports, and many PIC 
members are already feeling overloaded by the reams of 
performance data currently being generated and 
disseminated at the SDA level. As one PIC member 
pointed out: "There's a limit to how much any 
volunteer can read or digest on a regular basis. We 
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don't need more information; what we need is better 
analysis . 11 



PIC members will generally support special purpose 
assessments when there is a need for specific types of 
information. But SDA staff report that it is harder to 
gain support for sustained ongoing evaluation 
initiatives. Attempts to promote new evaluation 
initiatives that require substantial SDA-level 
investments must specifically address the concerns of 
PIC policymakers. PIC members will want to know how 
proposed evaluation models will assist thsra in policy 
development, planning, funding, and oversight, and in 
improving the quality of local programming. 
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